Public cloud has some bandwidth challenges with respect to downloading and uploading data in the gigabytes to terabytes when you have a very interactive workflow. Many CAE applications generate a large amount of data for each run.
There are many solutions to this problem, this post is to summarize what we have found works best with customers
Automate the download
TotalCAE Public cloud starts downloading the results from the cloud locally to your on-premise fileshare once the job has finished. For example, If a job finishes on the cloud at 11PM while the user is sleeping, the transfer starts to their workstation automatically, and will be ready to post process when they arrive to work in the morning without having to initiate the download.
From the engineers perspective, there was no download penalty as it was ready and waiting when they arrived at work.
Enable high speed parallel download tools.
TotalCAE utilizes a special download client that can parallelize downloads to speedup file transfer by breaking them into small parts and downloading those in parallel, using multiple cores. Normal download clients like SFTP are single threaded, download from start to finish, and are not optimized for large bulk data transfer over the internet.
The TotalCAE solution also utilizes local high speed data connections at global edge locations, to make data downloads fast to different regions via dedicated high speed links, even if they are utilizing a cloud data center that is not regional to them.
In one case a customer who had previously ruled out public cloud because it was taking hours to bring data back when they tried some other vendor, was able to get that data back in 10 minutes which made a usable workflow.
Utilize compression on model output
The less data you have to transfer, the faster your downloads will go. See our post on compression options to learn about both application aware compression, and HPC compression tools. A 50% reduction in the file size will 1/2 the time you wait to get the results back.
Pay for a high bandwidth direct connection to the cloud data center
Most IaaS providers that TotalCAE partners with have the ability to pay for a dedicated network link from your company data center to the cloud provider datacenter. While this can be a very expensive option, it does perform very well for large data transfers and is the best solution where money is no object, or if your company already has this type of connection in place for other cloud projects. On Azure this is called ExpressRoute , on AWS this is Direct Connect, and on Oracle OCI it is FastConnect.
Avoid transferring the data with Remote Viz
Avoiding data transfer is the ultimate bandwidth saver. Often an engineer may need to do a small model modification or can do the pre/post on the cloud with a remote visualization solution like TotalCAE Remote Viz or our integrated cloud desktops to avoid bringing all the results back.
For example, we have customers using STAR-CCM+ CFD solver that may need to make a small change to the mesh and re-restart the job. By using a cloud workstation, they avoid transferring 100GB back to their on-premises workstation to make a small edit and have to re-transfer the model back to solve.
In another case, we have some customers using Abaqus that post-process the ODB file with Python and extract what is needed to avoid bringing back the entire ODB file.