My search continues. Click edit to open the Services > SSH configuration screen. SCALE runs Kubernetes so no need to invoke anything as Kubernetes will restore its state on bootup. I can't even use a different VLAN as Scale can only have a single default IP. 1 minute read. I really love TrueNAS scale but my only big pain point is how slow the GUI is. This is the output from kube-system. GlusterFS in TrueNAS Scale Bluefin - Problem in Bitrot/scrubbing/healing. 793. 0. #5. I restarted my system a little bit into the resilvering process to see if that'd fix the kubernetes issue but my issues still persisted. Add a bridge, name it something like br0, select your main interface as a member and add the IP alias that you had on main interface. Prior upgrade, make sure your root user has the password enabled into Angelfish UI. Use Netwait to prevent starting all network services until the network is ready. SCALE allows Kubernetes to be disabled. Oct 25, 2021. Root access to the system from a remote client is never recommended. Kubernetes (and in a similar way, Docker Swarm) do clusters and load balancing of container workloads according to configured instructions. I'm on TrueNAS-SCALE-22. 0 and the Port to 3260, then click SUBMIT. The TrueCommand web interface provides single sign-on functionality and unified administration of users and TrueNAS. Right now SCALE uses docker as the container backend for k3s; but a couple years from now that might be CRI-O or something else. Do the CMD and args entries even work at. Last Modified 2023-08-31 13:31 EDT. This is the output from kube-system. Using Shared Host Paths with Safety Checks Disabled. This page is being rebuilt with notes about the latest TrueNAS SCALE nightly development versions. remove gpu. Go to System Settings > Failover. Name it "jellyfin". 10-Beta1 MB: ASUS P10S-I Series RAM: 32 GB CPU:. 1 and I have the following problem. 1) is. My apps like plex run great on client devices with no lag or responsiveness issues. Set the IP Address to 0. You're running kubernetes, docker labels are not the same as kubernetes labels. As usual you can read more and get started using SCALE by taking a look at the Developer notes page:TrueNAS Plugins use the FreeBSD native jails capability as well as some middleware (iocage) to integrate with FreeBSD’s package and Ports systems. Name it "jellyfin". 10-Beta1 64GB RAM 10th Generation Intel i7 Samsung NVME SSD 1TB, QVO SSD 1TB Boot from Samsung Portable T7 SSD USBC CASE: Fractal Define 7 running TrueNAS SCALE 23. However, as the saying goes, with great power (apps) comes great responsibility - and I have needed two TrueNAS SCALE reinstalls in the past week on my homelab setup purely due to failure of the. In kubernetes you abstract the container port from the network-facing port, as you may have more than one copy of a container on a given node, and so you define a kubernetes service in between which is like a loadbalancer. I moved my home assistant from a docker installation on a raspberry to trueNAS scale (using the truecharts app). 0. 00. It prevents the docker daemon from being run, if no App is installed. On TrueNAS CORE systems, go to System > Advanced and click SAVE DEBUG. The problem is that changing the TrueNAS Scale gateway changes the default IP for any containers. The document is arranged in a parallel manner to the TrueNAS web interface, beginning with the top panel and then descending through each. 1-1 CPU: Intel(R) Xeon(R) CPU E5-1660 v3 @ 3. 1. 10-Beta1 64GB RAM 10th Generation Intel i7 Samsung NVME SSD 1TB, QVO SSD 1TB Boot from Samsung Portable T7 SSD USBC CASE: Fractal Define 7 running TrueNAS SCALE 23. So much so, that they haven’t really created much documentation for it. 2. With hints found on TrueCharts' Discord, here and in a Kubernetes forum, I was able to move my previous config into the TrueCharts containers including ingress & traefik. Creating the Cluster. At least there are no pods to choose from when it prompts me for one on the shell menu for the Plex app. 250. Sirfucksworth Cadet. This displays two additional Passphrase fields to enter and confirm the. 0. isp. Apps App. Advanced Settings. 10-Beta1 64GB RAM 10th Generation Intel i7 Samsung NVME SSD 1TB, QVO SSD 1TB Boot from Samsung Portable T7 SSD USBC CASE: Fractal Define 7 running TrueNAS SCALE 23. Enter a name for the interface. Click Apply and then Test Changes. Just like TrueNAS CORE, TrueNAS SCALE is designed to be. g. Jan 26, 2022 #1 i put a NVIDIA gpu in my NAS, then removed it, now it still always appears in the GPU configuration section of apps/containers, though its correctly not showing up in Advanced. The one other thing I did was to ensure that the docker service was running, and would be started every boot. After installing the SCALE . Update opens an upgrade window for the application that includes two selectable options, Images. 40. Nov 3, 2021. 4, the problem, at least on my installation, has been fixed. The point of this step is to create a user with all necessary permissions that you will use to deal with all things related to Jellyfin. In order to access data storage systems, the Kubernetes CSI was released in 2018. ESXi and TrueNAS SCALE Supermicro X10SDV-4C-TLN4F mainboard Supermicro. 12. 3K. Please to announce that #TrueNAS SCALE 22. However, as the saying goes, with great power (apps) comes great responsibility - and I have needed two TrueNAS SCALE reinstalls in the past week on my homelab setup purely due to failure of the kubernetes/app system. To find an existing pool, log in to the web interface and go to Storage > Pools. This host could be either a VM running on TrueNAS SCALE itself or a Raspberry PI running in my network. KVM can't pass PCI and USB properly, apt-get is unavailable and so on. Select “more”. Version: TrueNAS CORE 13. I did a fresh install of TrueNAS Scale 22. Threads 6. This came along with some other issues, wich were there even present for at least two days berfore the reboot (and were in fact. for Truenas scale CLI. Case: Fractal Design Define 7 XL Power Supply: Corsair RM750X 80+ Gold Motherboard: Supermicro X11SPI-TF CPU: Intel Xeon Silver 4210T (10c/20t) Cascade Lake 2. The Kubernetes Settings screen allows users to customize network, system, and cluster settings for all apps in. To set up a bridge interface, from the Network screen: Click Add in the Interfaces widget. 55. TrueNAS will take you to the Two. A new implementation of the CSI is the Democratic CSI driver that connects Kubernetes, and other container systems, with the open source ZFS file system. I'm attaching screenshots to help explain the issue I'm having. I was able to get my NAS PC up and running after many restarts and resets of configuration settings. I have assigned specific IPs to those pods and work fine however, the FW still thinks they are sourcing from the host (TrueNAS SCALE), resulting in not routing through VPN. TrueNAS-SCALE-22. Minor update-. 0-U5 SCALE Cluster: 2x Intel NUCs running TrueNAS SCALE 23. Containers continue to work without k3s server, though the Apps web UI doesn't. #1 Can I possible disable k3s service and using docker cli directly? M morganL Captain Morgan Administrator Moderator iXsystems Joined Mar 10, 2018. #2. 12 release in the coming days. The TrueNAS CLI guide for SCALE is a work in progress! New namespace and command documentation is continually added and maintained, so check back here often to see what is new! Welcome to the TrueNAS SCALE Command Line Interface (CLI) guide! The TrueNAS CLI in TrueNAS SCALE functions like a text-based. Version: TrueNAS CORE 13. TrueNAS is ongoing, with each system bringing unique. The only way to enable/disable Kubernetes is to set or unset the storage pool, which doesn't actually completely stop/start Kubernetes from running, as far as I can tell. Explained here:. TrueNas Scale has a compelling Helm+Kubernetes-based application hosting solution for things you might want to self-host for personal life improvements. CronJob Configuration: Includes options to enable, configure, and schedule cron jobs as part of the application deployment. dirtyfreebooter Explorer. 2 GHz 95 W RAM: 3x 64 GB + 1x 32 GB DDR4 2400 ECC LRDIMM Extra HBA: Passthrough HPE H220 (LSI 9205-8i) - FW P20. I've found some guides on configuring Prometheus/Grafana for Kubernetes, but it doesn't match the Scale implementation and I'm too much of a beginner to see how to adapt the instructions. If an unavoidable critical situation requires allowing root access, it is recommended to configure two-factor authentication first. Advanced (Kubernetes) Settings. Version: TrueNAS CORE 13. Enterprise Applications: Tutorials for using TrueNAS SCALE applications in an Enterprise-licensed deployment. #6. 10-Beta1 MB: ASUS P10S-I Series RAM: 32 GB CPU:. 1:6443 ssl:default [Connection reset by peer] 2022-05-05 12:26:45 (Asia/Shanghai) Dismiss I did not see it and I. 134. When you are ready to create a container, go to Apps, click Discover Apps, then click Custom App. TrueNAS SCALE EULA; Software Development Life Cycle; TrueNAS Data Collection Statement; SCALE Tutorials ⎙ Download or Print; Main Dashboard. 3. TrueNAS SCALE is the latest member of the TrueNAS family and provides Open Source HyperConverged Infrastructure (HCI) including Linux containers and VMs. 10 minute read. I just restarted my system and it presented me this critical alert: Code: Failed to start kubernetes cluster for Applications: year 0 is out of range. 2 is now available! Lots of bugfixes and small improvements in this release. 2: I know that. Wrapping Up. Version: TrueNAS CORE 13. Services will monitor continuously the running Pods using endpoints, to ensure the traffic is sent only to available Pods. that would be too easy. The goal for my Scale rig is as follows (system details in sig): 10G NIC -> Dedicated to local access (GUI, SMB, etc) via my local storage VLAN. I just tried the upgrade to TrueNAS Scale 22. 250 (configured as Kubernetes NodeIP) Kubernetes. I restarted my system a little bit into the resilvering process to see if that'd fix the kubernetes issue but my issues still persisted. Click edit to open the Services > SSH configuration screen. It includes many new features and improved functionality that span initial effort for high availability (HA) feature support and improvements, and new or improved features in SCALE applications, services, ACLs, and shares: New Kubernetes Passthrough functionality in SCALE applications. 0 Added TrueCharts The first app (Tailscale) install yields the below. Quad Intel NIC. 0. Advanced Settings. Application Configuration. Dual Xeon E5645 processors (6 core - 12 threads X 2) @2. Sub-forums. All users should create the local administrator account and use this account for web interface access. The Global Configuration screen displays. 10-Beta1 MB: ASUS P10S-I Series RAM: 32 GB CPU: Intel(R) Xeon(R) CPU E3-1240L v5 @ 2. Gurus I'll cut to the case, we're all busy Clean install of TrueNAS-SCALE-22. TrueNAS SCALE was designed as a smart NAS with better support for Apps. Select the disk size from the list that displays. I'm on Scale 22. brando56894 said: The reason for the VM was just because the TrueNAS webUI takes over ports 80 and 443, and obviously my nginx container couldn't bind to those as well. morganL said: Need to describe what you are doing very specifically. My Minio set-up disappeared, but I can re-instate it. S. To configure network settings: Disable the failover service. To configure SSH, disable the service and click the edit. 0. . So zeroconf cannot bind the port 5353, and my device (a tahoma box) is not. 12. 0-U5 SCALE Cluster: 2x Intel NUCs running TrueNAS SCALE 23. To upgrade an app to the latest version, click Update on the Application Info widget. Kubernetes is the leading open source container management system. TrueNAS Core-13. If enabled on your network, TrueNAS uses DHCP to assign global network. TrueNAS SCALE single controller systems use the DHCP-assigned IP address for the primary network interface to. As a new TrueNAS SCALE user, I'm impressed by the power and flexibility of the software, and the excellent NAS features. Individual services have configuration screens and activation toggles, and you can set them to run automatically. 2. Debian 12 Bookworm was JUST RELEASED last month with kernel 6. 10-Beta1 MB: ASUS P10S-I Series RAM: 32 GB CPU:. The Kubernetes Settings screen allows users to customize network, system, and cluster settings for all apps in TrueNAS SCALE. 3. Show : Old Main NAS. Logging In as Admin. Click on the interface to open the Edit Interface screen for the selected interface. I am leaning towards Truenas Scale as it is free. 2,458. TrueNAS SCALE Enterprise customers should contact iXsystems support to obtain license information for their TrueNAS system. Setting up SMB Shares for Windows. iso file on both controller 1 and 2 and finishing the TrueNAS SCALE Installer process, use the Console setup menu to configure the required network settings on controller 1 so it can access the SCALE UI. Options are documented on this page as CLI flags, but can also be passed as configuration file options. Since I really don't want to disable my primary Ethernet port I'm hesitant to start changing things. Provides information about the app kubernetes namespace in the TrueNAS CLI. The updated software supports SMB, Gluster, and NFS File Sharing, iSCSI Block Storage, S3 Object API integration, and Cloud Sync for interoperability with public cloud storage. vlan10 – I want k3s on this subnet and no other TrueNAS Scale services. TrueNAS SCALE is scale-out storage and hyperconverged infrastructure that uses Kubernetes for deploying containerized (e. you can run a linux vm and run native docker in the vm, try the docker-compose. I'm attaching screenshots to help explain the issue. Press Enter after entering each IP address. After installing an SSD and upgrading to TrueNAS-SCALE-22. But Kubernetes still won't start. After generating the debug file, TrueNAS prompts you to download it to your local system and saves a copy in. Administrative user (admin), enter the username admin and password you set up. The reason I'm asking this is the fact that my Truenas freezes (completely unresponsive) on a daily basis (no data in the logs, etc) and I'm trying to test different settings and see if I can find the culprit. With hundreds of thousands of testers and contributors, the TrueNAS community development model enables broader testing, and ultimately, a higher quality product, in addition to its unbeaten value. Global Configuration Screens. Save this file in a safe location on your network where files are regularly backed up. Enter a name for the interface. 0 which I assume is correct since its locally hosted on the SCALE server. 10-Beta1 MB: ASUS P10S-I Series RAM: 32 GB CPU:. The Add Interface configuration screen displays. Disable ipv6 may be work . The point of this step is to create a user with all necessary permissions that you will use to deal with all things related to Jellyfin. 1 minute read. #2. Hey, I just can't get my head around Kubernetes. Replace tank/vm/haos with your ZVOL path. 0-U5 SCALE Cluster: 2x Intel NUCs running TrueNAS SCALE 23. ago. The current drivers implement the depth and breadth of the. Synchronizing System and. Route v4 interface: NIC2. Next, under the Installation media, you can select an existing ISO image file, or you can Upload New Image File which will open a dialog box to upload a new ISO. If not selected, only disks of the size selected in Disk Size are used. You don;t have to stick to those ports though,. Anything and everything to do with TrueNAS SCALE. TrueNAS Scale Interfaces: en02 is on vlan1. #3. Jun 28, 2021. The System Settings > Services > SSH screen allows you to set up SSH service on TrueNAS SCALE. #1. Guys , The latest update not working with GPU, we have to wait for fix, if you want to have the latest version installed, do as i said, disable gpu in kubernetes advanced settings. Last Modified 2023-08-31 14:46 EDT. This second update (22. . I am also planning to run lots of docker containers in them. Generally speaking you cannot use "the docker way" with Traefik on TrueNAS SCALE. I would suggest starting again and place the apps where you want them. 15. 11, but not 10. Advanced settings have reasonable defaults in place. Advanced settings have reasonable defaults in place. Note that servers also run an agent, so all of the configuration options listed in the k3s agent documentation are also supported on servers. Jun 17, 2021. As I've explained in another post/thread here, there is an issue with snapshots being created for ix-applications dataset and its child datasets. You can use democratic-csi documentation and achieve the same results but the reason I created this guide is the fact that democratic-csi docs are covering multiple awkward combinations of various. #1. Version: TrueNAS CORE 13. Expand the ‘Application Events’. 38. Add a new connection and change the setup method to Manual. It's Time to Kick the Tires. However, I'm looking for something more robust as I had problems with this in the. The user will then have access to the native container services within Debian. . 10-Beta1 MB: ASUS P10S-I Series RAM: 32 GB CPU: Intel(R) Xeon(R) CPU E3-1240L v5 @ 2. TrueNAS Scale is running on bare metal, and I have 4 physical NICS dedicated to it. Now each network must be 1 per entry field using the Add button AND the default cluster IP address set in the kubernetes settings is automatically added to the exclude list so you don't have to define it. In the TrueNAS GUI, go to [Credentials > Local Groups] and create a group. For TrueNAS Enterprise 13. 10-Beta1 64GB RAM 10th Generation Intel i7 Samsung NVME SSD 1TB, QVO SSD 1TB Boot from Samsung Portable T7 SSD USBC CASE: Fractal Define 7 running TrueNAS SCALE 23. 11. Messages. You cannot change the Type field value after you click Save. TrueNAS SCALE Enterprise customers should contact iXsystem Support for assistance with the initial set up and configuration. 10-Beta1 MB: ASUS P10S-I Series RAM: 32 GB CPU:. #43. TrueNAS SCALE 23. I also upgraded to 16GB of RAM. 0 still. Releases for major versions can overlap while a new major version is working towards a stable release and the previous major version is still receiving maintenance updates. 10-RC. Truenas-Scale (Bluefin) Supermicro A2SDi-H-TF, 64GB ECC RAM in Fractal Design R5 Intel DC3700 100Gb as boot disk Apps: 2 * Samsung SM863 960GB in Mirror Tank: 7 * Tosh MG09 18TB in Z2 + Intel M10 64GB Optane as L2ARC (Metadata only) UseCase: Backup via Replication of PrimaryNAS Power: 100WMay 19, 2021. 10-Beta1 64GB RAM 10th Generation Intel i7 Samsung NVME SSD 1TB, QVO SSD 1TB Boot from Samsung Portable T7 SSD USBC CASE: Fractal Define 7 running TrueNAS SCALE 23. brando56894 said: The reason for the VM was just because the TrueNAS webUI takes over ports 80 and 443, and obviously my nginx container couldn't bind to those as well. 0-U5 SCALE Cluster: 2x Intel NUCs running TrueNAS SCALE 23. At least there are no pods to choose from when it prompts me for one on the shell menu for the Plex app. the k3s stack (default SCALE docker settings) by default (which is what concerns OP) doesn't utilise iptables at all. I've just installed TrueNAS Scale (21. ipv6. A new implementation of the CSI is the Democratic CSI driver that connects Kubernetes, and other container systems, with the open source ZFS file system. morganL said: Need to describe what you are doing very specifically. I know I can change the VLAN for all apps in advanced Kubernetes settings, but I hoped it should also work per app. I have seen a few similar problems to the one i am about to describe, which have been solved, but i find that the solutions do not solve my problem. All Apps are OK. Jul 24, 2022. Using the first release of TrueNAS SCALE and TrueCharts they cannot. This feature is available on the nodes of your cluster. Can I simply deselect the kubernetes pool and the reselect it again when I want apps to start up or will deselecting the pool delete all the ix-applications datasets or wreck havoc in other ways? Jul 11, 2022. Version: TrueNAS-SCALE-22. As to be expected, none of my Apps are running. Intel Xeon E3-1220V3 - BX80646E31220V3. Release numbering will be based on Year and Month. Docker) applications. Some scenarios I've thought of - where sharing directories across apps and SMB shares makes sense to me* * Synchthing App: Allow access over SMB/NFS to the root dir where Syncthing stores a 'master copy' of files - mainly incoming pictures from a phone's camera. Click the in an app widget to see the list of app options, then select Delete. On TrueNAS CORE systems, go to System > Advanced and click SAVE DEBUG. This is cool, but how thing are evolving with TrueNAS and its container system is a bit at odds with how things are evolving with Docker and Kubernetes. Click PROCEED to generate the debug file (might take a few minutes). I could run a VM with the containers BUT then I get all the access permissions grief trying to get access to the Scale Storage Click on the interface to open the Edit Interface screen for the selected interface. TrueNAS SCALE Systems Microsoft Active Directory TrueCommand Container TrueNAS SCALE Systems Follow this procedure for each TrueNAS SCALE system you want to connect to TrueCommand and use in the cluster. 105. 10-Beta1 64GB RAM 10th Generation Intel i7 Samsung NVME SSD 1TB, QVO SSD 1TB Boot from Samsung Portable T7 SSD USBC CASE: Fractal Define 7 running TrueNAS SCALE 23. 0 from a new piece of install media created with a fresh install image to rule out the installer messing up, and also because statisfactory stopped functioning out of nowhere, and. 3K. 0-U5 SCALE Cluster: 2x Intel NUCs running TrueNAS SCALE 23. To upgrade multiple apps, click the Update All button on the Installed applications header. TrueNAS SCALE includes the ability to cluster systems and provide scale-out storage with capacities of up to hundreds of Petabytes. Now, with k3s disabled it's dropping to 15. 10-Beta1 MB: ASUS P10S-I Series RAM: 32 GB CPU: Intel(R) Xeon(R) CPU E3-1240L v5 @ 2. 0-U5 SCALE Cluster: 2x Intel NUCs running TrueNAS SCALE 23. Version: TrueNAS CORE 13. 12. It also hosts SMB shares and replicates TBs of data with less. TrueNAS SCALE 22. I'm excited to see Linux-based TrueNAS shipping with Docker and Kubernetes! I've been testing out the apps through the UI and began trying to convert my Docker Compose files to Helm then massaging those into TrueNAS-compatible charts. Allowing the validation check to. Hello TrueNAS community, Specs first: - 2 x GTX 1080TIs - Ryzen 9 5900X - 64Gb ram - ASUS Rog Crosshair VIII Hero TrueNAS-SCALE-22. role. The current focus is providing storage via iscsi/nfs from zfs-based storage systems, predominantly FreeNAS / TrueNAS and ZoL on Ubuntu. . You can run Tailscale inside a Kubernetes. that will give you 4. 0 these two bugs were fixed but sound strangely similar to my problem: - Kubernetes will not start on fresh install - SCALE Nightlies - Failed to start kubernetes cluster for ApplicationsEdit your main interface, remove the alias IP. Use complex passwords and Two-Factor Authentication ( CORE 2FA , SCALE 2FA ). 0. The way it works is. 2K. I updated. I receive the same error: " CRITICAL Failed to configure kubernetes cluster for Applications: Missing 'cpuset, cpu' cgroup controller(s) which are required for apps to function 2023-04-21 09:36:48 (America/Los_Angeles) "Stopping an application in the CLI: For the linux shell/ssh. This is the my version of TrueNAS Scale: TrueNAS-SCALE-Bluefin-RC - TrueNAS SCALE Bluefin RC. There will be a Container Storage Interface (CSI) that can couple the container services with the SCALE storage capabilities. I'm starting to conclude apps just don't work on a fresh install of Scale 22. In Bluefin, non-root local user UID starts with 3000, versus Angelfish UID 1000. Use the format bondX, vlanX, or brX where X is a number representing a. Advanced (Kubernetes) Settings. I really love TrueNAS scale but my only big pain point is how slow the GUI is. 17. ESXi and TrueNAS SCALE Supermicro X10SDV-4C-TLN4F mainboard Supermicro SCE300 chassis Intel Xeon D-1518 - 4 cores 32 GB ECC memory 1x Samsung SSD 860 EVO 1 TB (boot and datastore1) 1x Supermicro AOC-SLG3-2M NVME card 3x Samsung SSD 970 EVO Plus 256 GB (RAIDZ1 pool for SCALE) 3x Noctua NF-A4x20. I just upgraded my Truenas Core system to Scale (Bluefin release version) and everythign seems to have gone well. TrueNAS Scale 22. Is it possible to add this with the big blue button, I have tried several different entries but with all of them the container won't start. TrueCommand assists in managing TrueNAS systems through REST APIs, WebSocket APIs, and a web user interface. 10 and 10. From the official SCALE web: TrueNAS SCALE provides simple access to the well-established Linux container ecosystem and makes application deployment. Grummeltier said: Hi, I use TrueNAS-SCALE-22. 02. 0. 02. 1 family is coming from. Edit line 20 to point to the Dataset you created in stage 1. The “hostPathValidation” check is designed to prevent the simultaneous sharing of a dataset over a file. Have more questions or want to discuss. # 3 Edit line 20 of the script, set a path to. The Alert Settings screen displays options to create and edit alert services and to configure warning levels and frequencies. Last Modified 2023-08-31 13:31 EDT. Next, go to [Credentials > Local Users] and create a user named "jellyfin". A warning message displays for some settings advising of. I also can't disable/reenable SMB - I get a similar error: (sqlite3. Truenas virtual machine network configuration. To set up Plex, you need a reliable, high-performance hosting solution. Maybe a feature. Enter the MinIO S3 root user in Root User and the S3 password in Root Password. Update opens an upgrade window for the application that includes two selectable options, Images. 0 these two bugs were fixed but sound strangely similar to my problem: - Kubernetes will not start on fresh install - SCALE Nightlies - Failed to start kubernetes cluster for ApplicationsEdit your main interface, remove the alias IP. Upon Saving, Plex is just stuck in a Deploying State (Cant get past it) Last edited: Jan 10, 2023. In that case a VM or seperate machine is the way to go, because an Appliance isn't really suited for completely custom setups. 5 minute read. Click Export Password Secret Seed and then click Save. The Linux-based platform uniquely offers KVM virtualization, native container support, Kubernetes integration, and the ability to scale up to 1,200 drives and 25PB+ in. @SimoneF I've only looked at 20.