Skip to content

performance

2 posts with the tag "performance"

Large Cloud Transfers: When a Desktop App Is the Right Tool

Where a desktop app fits for large cloud transfers, versus a provider's own migration

Moving a few files is easy anywhere. Moving terabytes is a question of where the work runs, and the answer depends on the job. For most transfers, a desktop app on your own machine beats a browser-based or hosted tool. For a single bulk move of a very large dataset into one provider, that provider's own migration service can be faster. This page sorts out which is which.

A browser-based transfer happens inside a tab, and a tab is a small, supervised space. It has a memory budget, it can be paused or discarded when you switch away, and it usually holds only a handful of connections. For a dozen photos this is invisible. For a media library it is the wall the transfer hits: the tab slows, the laptop heats up, and a job that should run for an hour cannot stay awake long enough to finish.

The Timeout and the Bandwidth Bill

Section titled "The Timeout and the Bandwidth Bill"

Hosted services move work onto their own machines, which sounds like an upgrade until you look at the path. The data has to travel from the source into their infrastructure, and then back out to the destination. You wait for both legs, and the company pays for the bandwidth in the middle. That cost does not disappear. It comes back as a subscription, a per-GB charge, or a monthly transfer cap.

Server functions also run on a clock. AWS Lambda, for example, documents a maximum function timeout of 900 seconds, or 15 minutes[1]. Many serverless jobs are designed to finish in seconds or minutes, not the hours a large migration needs. When the clock runs out mid-file, the job has to be broken into pieces and stitched back together; if that orchestration is wrong, gaps and duplicates are the failure mode.

What a Desktop Process Can Actually Do

Section titled "What a Desktop Process Can Actually Do"

A desktop app is not boxed in the same way. It can use the memory and processor your machine actually has, open many transfers in parallel, and keep running for as long as the work takes. If the connection drops, Blober can retry failed records; if you stop and rerun a workflow, skip-existing keeps completed files from being sent again.

Blober also streams rather than stages. Instead of saving a full copy of every file to your disk and uploading it again, it flows the data through and writes it straight to the destination, so disk space stops being the limit. The difference between a streamed transfer and download-and-reupload is small for ten files and decisive for ten thousand, which is covered in Server-to-Server vs Download-and-Reupload.

Because the transfer runs from your machine, it uses your own connection and your own IP address. There is no shared relay address for a provider to throttle or block for every user of the tool at once. A hosted service, by contrast, reaches every provider from the same pool of company servers, which is the traffic pattern most likely to attract shared-service limits. Your own account and API limits still apply either way.

When a Provider's Own Migration Wins

Section titled "When a Provider's Own Migration Wins"

For a single move of a very large dataset, tens of terabytes or tens of millions of files, into one destination, the fastest path is often not your machine at all. Some providers run datacenter-to-datacenter migration, where the data never makes the round trip through your home or office connection. Backblaze, for example, lists assisted data migration as available for B2 Cloud Storage[2], and what can take weeks streaming through your own link can take far less over a datacenter one. If you are doing one big move into one provider and you can grant that service access, start there. There is a worked 25 TB example in Migrating 100M+ Files from DigitalOcean Spaces to Backblaze B2.

This is not a flaw in any one tool. Any client-side transfer, a desktop app or a command-line tool alike, has to pull the data down to your machine and push it back up, which is two trips through your connection. For the very largest one-time moves, skipping those trips is the whole point.

Where a Desktop App Is the Right Tool

Section titled "Where a Desktop App Is the Right Tool"

A provider migration helps in a narrow case: one big move, into one provider that offers it, when you can hand that provider access. Outside that case, a desktop app like Blober is usually the better fit, and for some teams it is the only acceptable one.

  • When you cannot share credentials with anyone. Assisted migration means granting a third party access to your storage. For regulated data, client data under NDA, or anything inside a SOC 2 or ISO 27001 boundary, that is often ruled out. A team in that position cannot hand its keys to a destination provider's migration service any more than it could hand them to a relay. Blober keeps your credentials on your own machine and adds no third party to the path, so the transfer stays inside the controls you already have. See How Blober Works.
  • When the move is ongoing, not one-time. Migration services are built for a single bulk lift. New uploads, folder moves, periodic refreshes, and re-running a saved job are what a desktop tool is for.
  • When no assisted path exists. Provider migration almost always moves data into that one provider. Moving between two arbitrary providers, or pulling a library out of a consumer cloud like GoPro Cloud or Google Photos, has no assisted option. A desktop app that speaks each provider directly is the practical route.
  • When the scale is moderate. For gigabytes or a few terabytes, arranging an assisted migration is more trouble than it is worth. A desktop app just runs.
  • When you want to watch it happen. A desktop transfer stays visible and under your control, so you can pause, resume, and verify on your own machine.

What a Desktop App Does and Does Not Solve

Section titled "What a Desktop App Does and Does Not Solve"

A desktop app is not magic, and it is not always the fastest option. The bytes travel over your own connection, so a slow link is still slow, and for the very largest one-time moves a provider's datacenter migration can beat it outright. What a desktop app gives you is control and privacy: no relay, no third party holding your keys, and the reach to move between any providers you connect, including the ones with no migration service at all.

Is a desktop app the fastest way to move a very large dataset? Not always. For a one-time move of tens of terabytes into a single provider that offers datacenter migration, that route is usually faster, because it skips the trip through your connection. A desktop app wins on control, privacy, ongoing use, and moves between providers that have no assisted option.

What if I cannot give a third party access to my storage? Then a desktop app is the right choice. Blober uses your credentials on your own machine and adds no outside party to the transfer, so it fits inside compliance boundaries that rule out both hosted relays and assisted migration services.

Does the data still go through my computer? Yes. A desktop tool streams the data through your machine to the destination. The point is that it does not also route through a separate company's servers, and it does not save a second full copy to your disk.

What happens if my transfer is interrupted? Blober retries failed records during a run. If you stop and rerun the workflow, skip-existing avoids sending files already written at the destination, so the job continues at file level instead of starting the whole migration again.

Upload to GoPro Cloud 4x Faster with Parallel Transfers

Upload to GoPro Cloud 4x faster with Blober parallel uploads

If you shoot with a GoPro, you know the routine. After a day of riding, diving, or traveling, your camera has anywhere from 20 to 200 GB of footage. You plug your camera in or connect via Wi-Fi, and the GoPro app starts uploading to GoPro Cloud.

The upload is slow. Not because your internet is slow, but because GoPro's app sends files one at a time. It picks a file, uploads it, waits for confirmation, then starts the next one. If you have 100 clips from a weekend trip, each one sits in a queue while the previous clip finishes.

This sequential approach means you are never using your full upload bandwidth. Most internet connections can handle several simultaneous uploads. A connection with 50 Mbps upload speed could be pushing four or five files at once, but GoPro's app uses it for just one.

How Blober Handles GoPro Cloud Uploads

Section titled "How Blober Handles GoPro Cloud Uploads"

Blober uses parallel upload streams when transferring files to GoPro Cloud. Instead of sending one file at a time, Blober opens multiple concurrent connections and uploads several files simultaneously.

The result is straightforward: if your connection can handle four simultaneous uploads (and most can), you finish in roughly a quarter of the time.

This is not a theoretical number. It comes down to basic network utilization. GoPro's sequential uploads leave bandwidth idle between files and during handshake overhead. Blober keeps the pipe full by starting the next upload before the previous one finishes its server-side confirmation.

ScenarioGoPro AppBlober
50 GB weekend trip (100 clips)~4 hours~1 hour
120 GB week-long shoot (300 clips)~10 hours~2.5 hours
8 GB quick session (15 clips)~35 min~8 min

Times based on a typical 30 Mbps upload connection. Actual speeds depend on your connection, file sizes, and GoPro Cloud server conditions.

GoPro's web interface limits downloads to 25 files at a time. If you want to download 500 clips, you need to repeat the process 20 times.

Blober has no batch limit. Select 10 files or 10,000 and start the transfer. Blober works through the entire queue without stopping to ask you to select the next batch.

Uploads fail. Connections drop, laptops go to sleep, Wi-Fi switches networks. When a GoPro Cloud upload fails through the app, it often starts the file over from scratch.

Blober tracks progress per file. If a transfer is interrupted, Blober resumes from where it stopped. For large files (GoPro's 5.3K videos can easily be 5-10 GB each), this saves real time. You do not re-upload 4 GB of a 5 GB file because your connection dropped at 80%.

Upload From Anywhere, Not Just Your Camera

Section titled "Upload From Anywhere, Not Just Your Camera"

GoPro's app expects you to upload from the camera or phone. If your footage is on an SD card, a NAS, or already in another cloud provider, you cannot use the app to get it into GoPro Cloud.

Blober lets you upload to GoPro Cloud from any source it supports:

  • Local drives and SD cards: Import footage from your card reader directly to GoPro Cloud
  • NAS devices: Upload from Synology, QNAP, or any network drive
  • Other cloud providers: Move files from Dropbox, Google Drive, or AWS S3 into GoPro Cloud
  • DJI or Insta360 footage: Consolidate action camera media from multiple brands in one cloud

Some people ask why you would upload to GoPro Cloud at all. Fair question. Here is when it makes sense:

  • You already have a GoPro Plus subscription and want to use the cloud highlight reels and editing features
  • You want automatic camera-to-cloud backup but need a faster way to bulk-upload existing footage
  • You shoot with DJI or other cameras and want all your action footage in one place with GoPro's editing tools

If you are moving away from GoPro Cloud, Blober handles that too. Transfer your entire library to Backblaze B2, AWS S3, Dropbox, or a local drive.

Blober is a one-time purchase with a lifetime license. No subscription, no per-GB charges.

Download Blober at blober.io