15-09-2021

Trials In Tainted Space Time Limit

A play through of Trials in Tainted Space where I show the basics of the game mechanics, story, combat, and general information for new players.

CPU limits is the maximum CPU time a container can uses at a given period (100ms by default). The CPU usage for a container will never go above that limit you specified. Kubernetes use a mechanism called CFS Quota to throttle the container to prevent the CPU usage from going above the limit. That means CPU will be artificially restricted. And remember, if you have been vaccinated, it takes time for protection to build. No vaccine is 100% effective and we still do not know how long immunity will last. You must be 18 years old to visit this site. Please verify your age. Trials in Tainted Space was made by fenoxo and you can play it here! Limit bot activity to periods with less than 10k registered users online. Server Time: Dec 7, 2020 08:29 PM.

At Buffer, we’ve been using Kubernetes since 2016. We’ve been managing our k8s (kubernetes) cluster with kops, it has about 60 nodes (on AWS), and runs about 1500 containers. Our transition to a micro-service architecture has been full of trial and errors. Even after a few years running k8s, we are still learning its secrets. This post will talk about how something we thought was a good thing, but ended up to be not as great as we thought: CPU limits.

CPU limits and Throttling

It is s general recommendation to set CPU limits. Google, among others, highly recommends it. The danger of not setting a CPU limit is that containers running in the node could exhaust all CPU available. This can trigger a cascade of unwanted events such as having key Kubernetes processes (such as kubelet) to become unresponsive. So it is in theory a great thing to set CPU limit in order to protect your nodes.

CPU limits is the maximum CPU time a container can uses at a given period (100ms by default). The CPU usage for a container will never go above that limit you specified. Kubernetes use a mechanism called CFS Quota to throttle the container to prevent the CPU usage from going above the limit. That means CPU will be artificially restricted, making the performance of your containers lower (and slower when it comes to latency).

What can happen if we don’t set CPU limits?

We unfortunately experienced the issue. The kubelet , a process running on every node, and in charge of managing the containers (pods) in the nodes became unresponsive. The node will turn into a NotReady state, and containers (pods) that were present will be rescheduled somewhere else, and create the issue in the new nodes. Definitely not ideal isn’t it?

Discovering the throttling and latency issue

A key metric to check when you are running container is the throttling . This indicate the number of time your container has been throttled. Interestingly, we’ve discovered a lot of containers having throttling no matter if the CPU usage was near the limits or not. Here the example of one of our main API:

You can see in the animation that the CPU limits is set at 800m (0.8 core, 80% of a core), and the peak usage is at most 200m (20% of a core). After seeing, we might think we have plenty of CPU to let the service running before it throttle right? . Now check this one out:

You can notice the CPU throttling occurs, even though the CPU usage is below the CPU Limits. The maximum CPU usage isn’t even near the CPU limits.

We’ve then found a few resources(github issue, Zalando talk, omio post) talking about how throttling lead to poorer performances and latency for your services.

Why do we see CPU throttling while CPU usage is low?The tldr is basically a bug in the Linux kernel throttling unecessarly containers with CPU limit. If you’re curious about the nature of it, I invite you to check Dave Chiluk’s great talk, a written version also exists with more details.

Removing CPU limit (with extra care)

After many long discussions, we’ve decided to remove the CPU limits for all services that were directly or indirectly on the critical path of our users.

This wasn’t an easy decision since we value the stability of our cluster. We’ve experimented in the past some instability in our cluster with services using too much resources and disrupting all other services present in the same node. That time was a bit different, we understood more about how our services needed to be and had a good strategy to roll this out.

How to keep your nodes safe when removing limits ?

Isolating “No CPU Limits” services:

In the past we’ve seen some nodes going to a notReady state, mainly because some services were using too much resources in a node.

We’ve decided to put those services on some specific nodes (tainted nodes), so those services will not disrupt all the “bounded” ones. We have better control and could identify easier if any issue occurs with a node. We did this by tainting some nodes and adding toleration to services that were “unbounded”. Check the documentation on how you can do that.

Assigning the correct CPU and memory request:

The main worry we had was service using too much resources and leading to nodes becoming unresponsive. Because we now had great observability of all services running in our cluster (with Datadog), I’ve analyzed a few months of usage of each service we wanted to “unbound”. I’ve assigned the maximum CPU usage as the CPU request with a > 20% margin. This will make sure to have allocated space in a node. If k8s won’t try to schedule any other service in that node.

You can see in the graph that the peak CPU usage was 242m CPU core (0.242 CPU core). Simply take this number and make it a bit higher to become the CPU request. You can notice that since the service is user facing, the peak CPU usage matches peak traffic time.

Do the same with your memory usage and requests, and you will be all set!To add more safety, you can use the horizontal pod autoscaler to create new pods if the resource usage is high, so kubernetes will schedule it in nodes that have room for it. Set an alert if your cluster do not have any room, or use the node austoscaler to add it automatically.

The downsides are that we lose in “container density”, the number of containers that can run in a single node. We could also end up with a lot of “slack” during a low traffic time.You could also hit some high CPU usage, but nodes autoscaling should help you with it.

Results

I’m happy to publish really great results after few weeks of experimentation, we’ve already seen really great latency improvements across all the services we’ve modified:

The best result happened on our main landing page (buffer.com) where we speed the service up to 22x faster!

Is the Linux kernel bug fixed?

The bug has been fixed and merged into the kernel for Linux distribution running 4.19 or higher (kudo again to Dave Chiluk for finding and fixing that).

However, as for September 2nd 2020, when reading the kubernetes issue, we can see various Linux projects that keep referencing the issue, so I guess some Linux distribution still have the bug and working into integrating the fix.

If you are below a Linux distribution that has a kernel version below 4.19, I’d recommend you to upgrade to the latest Linux distribution for your nodes, but in any case, you should try removing the CPU limits and see if you have any throttling. Here a non exhausting list of various managed Kubernetes services or Linux distribution:

  • Debian: The latest version buster has the fix, it looks quite recent (august 2020). Some previous version might have get patched
  • Ubuntu: The latest version Ubuntu Focal Fosa 20.04 has the fix.
  • EKS has the fix since December 2019. Upgrade your AMI if you have a version below than that
  • kops: Since June 2020, kops 1.18+ will start using Ubuntu 20.04 as the default host image. If you’re using a lower version of kops, you’ll have to probably to wait the fix. We are currently in this situation.
  • GKE (Goggle Cloud) : The kernel fix was merged in January 2020. But it does looks like throttling are still hapenning

ps: Feel free to comment if you have more precise information, I’ll update the post accordingly

If the fix solved the throttling issue?

I’m unsure if totally solved the issue. I will give it a try once we hit a kernel version where the fix has been implemented and will update this post accordingly. If anyone have upgrade I’m keen to hear their results.

Takeaways

Trials In Tainted Space Time Limited

  • If you run Docker containers under Linux (no matter Kubernetes/Mesos/Swarm) you might have your containers underperforming because of throttling
  • Upgrade to the latest version of your distribution hoping the bug is fixed
  • Removing CPU limit is a solution to solve this issue, but this is dangerous and should be made with extra-care (prefer upgrading your kernel first and monitor throttling first)
  • If you remove CPU limits, carefully monitor CPU and memory usage in your nodes, and make sure your CPU requests are
  • A safe way to is to use the Horizontal pod autoscaler to create new pods if the resource usage is high, so kubernetes will schedule it in nodes that have space.

👉Hacker news update: lot of insighful comments. I’ve updated the post to have better recommendations. You should prefer upgrading your kernel version over removing the CPU limits. But throttling will still be present. If your goal is low latency, remove CPU limits, but be really mindful when doing this: set the proper CPU requests, add the necessary monitoring when you do this. Read the comment written by Tim Hockin from Google (and one of Kubernetes creator)

I hope this post helps you get performance gains on the containers you are running. If so, don’t hesitate to share or comment with always some insighful comments

Special thanks to Dmitry, Noah and Andre that adviced me on this.

Next reads:

👉 Why you should have a side project

👉 How we share technical knowledge in a remote team, across timezones

Trials In Tainted Space Time Limits



While this page is primarily used for reference, do note that, as a player, you are responsible for your actions in the game--this includes using cheats and exploits. It is highly advised to make a save before attempting any of the following cheats.

Always remember to keep a backup copy of your original, unaltered, save file!

Description

Trials in Tainted Space's gameplay is not immune to exploitation. There are various features the player can access, intentional or otherwise, to gain a desired advantage in the game. Most of these exploits may have been left in the game during various points of development and are subject to change if and while the game is in development.

This page will only list cheats that are acknowledged by the developers themselves and are approved for public release. The cheats listed will only be ones that are attainable in-game, during play. For out-of-game game-editing references, try the save editing page.


Cheat Codes

The game has various cheat codes that can be inputted by the player. This can be done in two ways:

  • By using a keyboard and directly typing the message at any time while playing the game.
  • By accessing a text input box (such as using V-KO's CustomInput option) and inserting and submitting the code there. The input box is limited in what cheats it can activate (so not all cheat codes are compatible with it), and is primarily there for users who use virtual, screen-prompted keyboards.
  • It is recommended to input cheat codes in on the codex console under options to prevent any unwanted actions such as moving or item usage.
  • Codes that cannot activate via the console can be activated in the main Codex screen. Because all of the main Codex options are assigned to numbers 1-5, this will prevent unwanted actions.
  • Holding Shift while typing will also prevent unwanted moving or item usage.

Trials In Tainted Space Time Limited Edition

This information is a bit outdated, so some codes may not work in current versions. Try at your own risk. link to a statement from Jacques00 on the forums

The cheat codes are as follows:

Cheat Codes
Input Result
treatment Forces the Treatment to use the default effects. (Does nothing if none of the other Treatment cheats have been activated).
bimbo Forces the Treatment to use the Female effects.
bull Forces the Treatment to use the Male effects.
amazon Forces the Treatment to use the Amazon effects.
cumcow Forces the Treatment to use the Cum-Cow effects.
fauxcow Forces the Treatment to use the Faux-Cow effects.
motherlode Gives enough XP to reach the next level.
88mph Pass large amounts of time.
urta (previously known as 'furfag') Enables infinite item use for most consumable items (Enter again to disable).
poison (previously known as 'sjw') Replaces “shemale” with “dick-girl” throughout the game’s scenes dynamically (since v0.8.010).
tistheseason Brings up a menu with a list of special holidays. The player may choose to toggle Seasonal events for that holiday to: Always On, Always Off, or Automatic (default).
marcopolo Unlocks all planets (excluding Breedwell and Kashima).
laplove Reverts the game to its previous inclusion of the Lapinara Parasitic Female and Prophylactic (Lapinara).
beshineforever
Opens the Doctor Badger Cheat Menu, which contains three separate sections:
  • Doctor Badger:
    • Options for Doctor Badger's interaction:
      • The state of either being active or arrested.
      • The state of either being pre or post bimbofication.
    • For the player:
      • Whether or not the silicone tank is installed in the player's ship.
  • Nym-Foe:
    • Options for Nym-Foe's combat:
      • The state of being broken, fixed, fixed and hostile, or not yet encountered.
      • The distribution of silicone between the player's breasts, butt, hips, and lips during a successful injection attack.
      • Whether Nym-Foe's 'Bouncy!' attack block will perform normally, block 100% of the time, or avoid being used.
      • The ability to add or subtract Nym-Foe's starting lust between 0 and 100, and the ability to reset back to 50.
    • For the player:
      • Add a Nym-Foe Dakimakura to the player's inventory.
      • Toggle whether or not the player has the Implant-tasticPerk.
  • Doll Maker:
    • Options for Doll Maker's combat:
      • The state of being disassembled, met and non-hostile, activated and hostile, or not yet encountered.
      • Whether Doll Maker's attack pattern will consist of its full moveset, only stat-effecting tools (if not applicable, choosing Bio-Magnetic Vibes instead), or only with its Bio-Magnetic Vibes.
      • How many points each of the stat-effecting tools in Doll Maker's moveset will subtract, being able to be set from 1 to 50 and reset to 5.
      • Whether or not the player starts the fight already cuffed.
    • For the player:
      • Toggle whether or not the player has the True DollPerk.
insaneinthemembrane Changes Bess' name to Runa.
mitzi Allows the player to instantly rescue Mitzi, bypassing the Stellar Tether Dungeon.
backinbizzness Resets Bizzy's interactions to before the player received her first e-mail. (added during version 0.8.066 development to showcase new scenes with Mitzi and Sera, continued availability past this version is not guaranteed).
doll Teleports the player to the start of the Kiro Quest dungeon (added during version 0.8.023 development, will most likely be removed at a later date or upon full quest implementation).
kqvrgoodness
Adds 5 virtual reality programs to the PC's inventory from or related to Kiro Quest. Once used, these allow the PC to access the fights (and corresponding sex scenes) in the Mindwash Visor to:
  • The fuckdolls
  • The Advanced Herm Sexbot
  • The Taursuit
healthdelivery Summons Bianca if possible (When on a hostile square where Bianca can spawn, and she is not in a bar).
anofferyoucantrefuse Skip to a certain part of Akane's first quest.
clowncar Allows ship flight with any amount of crew members.
anotherbrickinthewall Immediately gain 100K credits. Reusable. If entered outside the Console, credit count will be updated after any changing of screen. If entered with the Console, it must be exited for credits to be added. Consecutive uses without leaving the Console count as one activation and only add 100K credits.
UP+DOWN+LEFT+RIGHT Use the arrow keys on the keyboard to do this. Gives the player 5xZil Ration. You can get 10,000 Credits per full inventory by selling to Burt on Mhen'Ga. You can quickly sell a full inventory by holding down the '1' button.
party Starts or resets the Zaibatsu party. At the end of the party, Steele is placed at the Gyre Taxi stand on Dhaal. If Steele's ship is not already on Dhaal, it won't be at the Spaceport, making leaving impossible without other cheats.
wargiiandbess Resets the Siege of the Korg’ii Hold quest progress, along with any progress in Eitan's relationship. 0.8.116 patchnotes suggested that the cheat would get a name change, however that hasn't happened yet as of 0.8.124.
nastiness Resets the Nastizia event chain to Pre-invite.

Developer Codes

Not for inexperienced or uninformed players. Using any of these codes can seriously break quests and/or saves. As a player, you are responsible for your actions in the game. It is highly advised to make a save before attempting any of the following cheats.

Always remember to keep a backup copy of your original, unaltered, save file!


Space


The dev codes are as follows:

Developer Codes
Input Result
impulseBrings up Impulse menu for scene ID.
idclevBrings up the idclev menu for teleportation by room ID.
fuckyou Toggles debug mode.

Trials In Tainted Space Time Limitations


Exploits

This is a list of exploitable events, items, or characters that the player can use to Captain Steele's advantage.

The exploits are as follows:

Exploits
Input Result
Dr. Lash Defeating him in combat will earn Steele 999999 credits and a significant amount of XP. This combat encounter can be repeatable if Steele continues to flirt with him. This can be done by stacking resistance to the burning element to or above 100%, while raising electric resistance as high as possible to minimize his damage output.
Syri Infinite Intelligence gain Holding down the 1 button during Syri's morning menu causes continuous slow intelligence gain

When combined with ClearYu can also give you Reflex, Aim, and Libido at the cost of Taint

Blackjack free money In the Treasure Nova casino on Zheng Shi, the player can place a custom bet up to their entire wallet balance. This, in conjunction with saving and loading, allows the player to acquire high amounts of credits quickly and with no risk.
Retrieved from ‘https://wiki.smutosaur.us/TiTS/index.php?title=Cheat_List&oldid=38440’