Skip to main content

Instant GitHub Codespaces are dangerous without realistic data

Picture the scene:

  • You click a button on a GitHub repository
  • A new tab launches with VS Code containing your code
  • All of the prerequisites are preinstalled and you can start coding, compiling, and running your application instantly

Amazing! Forget about walking line-by-line through “SETUP.md” to configure your local machine.

But let's fast-forward a little bit more…

  • You start your web app and load it in a new tab
  • You log in to your web app
  • You’re greeted with an empty screen

There’s no data. There are no other users. There’s no realistic data in this application at all.

Codespaces with databases

With GitHub Codespaces you can set up a cloud-hosted, containerized VS Code environment. You can then connect to a codespace through the browser or through VS Code.

The main question we are trying to answer now is:

Can we have virtualized environments with databases created from backups or scripts, not just empty databases, or fake data sets?

Update July 2021

Hello from the 🛸 Spawn 🛸 team!

Virtual environments with databases#

The Spawn team has been busy looking at how databases can fit with virtual environments. We've written articles about Github Codespaces, Wunderpreview and Gitpod. We are interested in how these different services can fit with Spawn to enable you to have virtual environments that can have GBs of realistic data as well. Have a read through our blog posts below.

Collaboration with Wunderpreview#

We are looking for people who want to work with us and Wunderpreview to set up virtual environments with data for you and your team. Wunderpreview is a company that helps you get a live preview environment for every new feature – fully automated. If you have a project where you’d like to create preview environments with data, then send us an email at spawn@red-gate.com or contact us via slack.

Exploring new pricing models#

Many of you have expressed interest in finding out how Spawn will be priced going forward. We’ve now started exploring various options when it comes to pricing models, please head to our pricing page. For now, if you’re already a Spawn user, this won’t affect you, but feel free to get in touch if you have any questions or thoughts!

Try the beta#

Spawn is currently in open beta, and free to use. It takes less than fives minutes to get up and running. Click the link below to join our beta 👇

👉 Get started

📝 Blogs#

Codespaces with databases#

With GitHub Codespaces you can set up a cloud-hosted, containerized VS Code environment. You can then connect to a codespace through the browser or through VS Code. The main question we are trying to answer in this article is:

Can we have virtualized environments with databases created from backups or scripts, not just empty databases, or fake data sets. Read more

Pull request previews with WunderPreview and Spawn#

Level up your pull request workflow with live preview environments backed by dedicated databases.

WunderPreview provides all stakeholders in a software project with a window into the work in progress features of an application. By building your application on every pull request and publishing it for all team members to see, visibility is increased and feedback is given earlier, tightening the build, test, deploy cycle. Read more

Going all-in on cloud-based development with realistic databases (with GitPod)#

Throughout 2020 and 2021 much of the world moved to remote-first working through unfortunate necessity. We got used to remote standups, meetings, reviews and collaboration. Working patterns and practices changed, but have development environments kept up? We’ll try to answer that using GitPod and Spawn. Read more

✨ New features#

🤖 spawn#

  • We have a brand new website to help new visitors understand Spawn better
  • Make data image sizes display correctly on app.spawn.cc
  • Fix container creation date display issues on app.spawn.cc
  • Improve support for MySQL
  • Improve overall reliability

⬆️ Backups#

  • Improve our upload strategy to better manage creating images from large backup files
  • We are dropping support for MongoDB backups

📖 Documentation#

  • Add search functionality to our doc site

Update June 2021

Hello from the 🛸 Spawn 🛸 team!

MySQL 8.0 support!#

We’ve been working on supporting MySQL 8.0 which is now available for you to use! Either restore your own database or use one of our public images.

New Github Actions!#

We've also created a set of Github Actions that you can use in your CI pipelines:

Raspberry Pi (ARM-based Linux) support!#

We thought Spawn on a Raspberry Pi would be cool, so we’ve officially added support for ARM-based Linux operating systems! Want to give it a shot? Just follow the standard installation docs.

Service Information#

We’ve published details of Spawn’s hosted environment and security measures to protect your data. Head over to our service information page to read more about it.

Try the beta#

Spawn is currently in open beta, and free to use. It takes less than fives minutes to get up and running. Click the link below to join our beta 👇

👉 Get started

📝 Blogs#

The key to great database Continuous Integration is disposable databases#

Database Continuous Integration (CI) is often the final piece of the CI story for a project and with good reason; it is hard to do and much harder to have something your team will describe as ‘great’. Read more

Development databases in Docker aren’t good enough#

Development databases in Docker aren’t good enough on their own. Why? Because they’re almost always so far from the production environment characteristics that you get a false sense of security in development. Read more

Simple Database CI with Spawn and Github Actions#

Spawn’s new Github actions make database CI easier than ever before. Read more

Going all-in on cloud-based development with realistic databases#

Throughout 2020 and 2021 much of the world moved to remote-first working through unfortunate necessity, but have development environments kept up? Read more

✨ New features#

🤖 spawnctl#

  • spawnctl proxy data-container will configure a localhost proxy on your machine on the default port for the database engine your data container is running on. This can be useful for environments where you've already got localhost connection strings configured and you don't want to manually update the port when data containers are created.

📡 app.spawn.cc#

  • We are in the process of enabling users to see their resources in the browser by visiting app.spawn.cc This is now the place to accept our EULA

⬆️ MySQL 8.0#

  • We now support MySQL 8.0. For a complete list of versions supported by Spawn, please see here. Create data-images with MySQL 8.0 by specifying version: 8.0 in your MySQL source file

🌤 Spawn Service#

  • Improve overall reliability
  • Various bugs fixes
  • Improve our observability of the system
  • Raspberry Pi (ARM-based Linux) support

📖 Documentation#

  • Created new login flow
  • Move our documentation to docs.spawn.cc
  • Various documentation improvements

Simple Database CI with Spawn and Github Actions

Running tests against databases in CI pipelines is an essential part of testing your application. 

Provisioning databases in CI pipelines can be hard work, however. Broadly speaking you have two options:

  • Have all your pipelines use shared databases
  • Use Docker to run containerised database instances

The first option has the advantage that you can test against real data, perhaps a recently restored copy of production, but it effectively serializes your pipelines as they contend for the shared database. You may be able to scale by adding multiple database servers, but ultimately the parallelism of your CI pipelines is limited by the number of database servers you have available to the pipelines.

The second option is a substantial improvement in terms of parallelism as each pipeline run now has a dedicated database spun up and torn down for exclusive use. However, the problem of testing against realistic data is now more acute. Typically where we see Docker being used to provision databases in CI pipelines, we see the use of seed data stored in the code repository used to populate the containerised database. This means you lose the confidence that you gain from testing against a realistic data set. If you don't want to go down the route of using seed data, you need to manage a docker volume inside your pipeline, or run a lengthy database restore operation in each pipeline run.

Development databases in Docker aren’t good enough

Development databases in Docker aren’t good enough on their own. Why? Because they’re almost always so far from the production environment characteristics that you get a false sense of security in development.

Having isolated databases is far better than a shared environment where other developers trample over your changes. But because dev databases tend to either be empty, or have “happy path” data within them, they never truly demonstrate the behaviours you’ll end up seeing in production.

This leads to a variety of different problems:

  • Unexpected data loss during schema migrations
  • Unacceptable latency on specific queries because of vastly different data sizes
  • Poor UX due to unanticipated user-provided data
  • UI glitches or performance issues not caught in lower environments because of unrealistic data
  • Entire branches of code left unexercised due to conditions on the data not caught in lower environments

Update May 2021

Hello from the 🛸 Spawn 🛸 team!

Migration testing#

We’ve been working on supporting Microsoft SQL Server 2019 which is now available for you to use! Either restore your own database or use one of our public images.

spawn.cc has had some changes - you can now find all Spawn documents over at docs.spawn.cc, where you can also view all previous newsletter updates.

We’ve recorded a video to demonstrate creating a MySQL database from a backup using Spawn - head on over to our YouTube channel to see MSSQL and Postgres examples too.

Try the beta#

Spawn is currently in open beta, and free to use. It takes less than fives minutes to get up and running. Click the link below to join our beta 👇

👉 Get started

🎬 Videos#

Creating Spawn Data Images from a MySQL backup#

Martin Podlubny, Software Engineer on the Spawn team takes us through how to create a Spawn data image from a MySQL Server backup.

Automated Database Testing: How Flyway, GitHub Actions and Spawn solve Azure migration#

Changing production databases can be risky, and the earlier you can catch problems the better. Learn how to use Flyway, GitHub Actions and Spawn, a new solution that provides database copies on demand, to easily test database migrations. Instead of relying on last-minute staging environments, Grant and Santiago will show you how to quickly automate migration tests against a fully populated database.

✨ New features and improvements#

🤖 Spawnctl#

💾 Backups#

⬆️ SQL Server 2019#

  • We are now supporting SQL Server 2019. For a complete list of versions supported by Spawn, please see our engine support page. Create data-images with MSSQL 2019 by specifying version: 2019 in your MSSQL source file.

🌤 Spawn Service#

  • Improve overall reliability
  • Upgrade the specs of our master nodes
  • Increase the spec of k8s nodes that hold data-containers
  • Improve our observability of the system

📖 Documentation#

  • Migrate from docusaurus v1 to v2
  • Move our documentation to docs.spawn.cc
  • Various documentation improvements
  • Added previous spawn updates to our docs blog

Update April 2021

Hello from the 🛸 Spawn 🛸 team!

Migration testing#

This month, we’ve been focusing on making it easier for you to get up and running with Spawn. We have amended our documents, and filmed videos to show you how easy it is to get a Spawn container running from your PostgreSQL or SQL Server database. A Spawn container is quickly provisioned with your database’s schema and data, and is perfect for development or testing against.

We also discuss the pitfalls of not testing your migrations before they reach your production code, and demonstrate them in this open source repository. To learn more, have a read of this Flyway blog we published.

If you’re already convinced of the importance of testing your database migrations, jump straight to our step-by-step guide on implementing these tests in your CI pipeline using Spawn. Alternatively, contact us directly and we’d be happy to help you get set up with migration testing in CI.

Try the beta#

Spawn is currently in open beta, and free to use. It takes less than fives minutes to get up and running. Click the link below to join our beta 👇

👉 Get started

📝 Blog posts#

Why you should be testing Flyway migrations in CI#

Bad migrations slip through to the release process despite our best efforts. Thankfully, there are solutions to this...

Read more about how to test Flyway migrations in CI with Spawn

🎬 Videos#

Creating Spawn Data Images from Postgres backups#

Chris Heppell, Lead Software Engineer on the Spawn team takes us through how to create a Spawn data image from a Postgres backup.

Creating Spawn Data Images from a SQL Server backup#

Martin Podlubny, Software Engineer on the Spawn team takes us through how to create a Spawn data image from a Microsoft SQL Server backup.

✨ New features#

🤖 Spawnctl#

  • You can now see which image ID a container was created from by running spawnctl get data-container -o wide or -o json or -o yaml
  • Running spawnctl version from docker image now reports the correct version
  • Bug fixes and improvements

💾 Backups#

⬆️ Postgres 11, 12 and 13.2#

We now support PostgreSQL 13.2. For a complete list of versions supported by Spawn, please see our engine support page. Create data-images with Postgres 13.2 by specifying version: 13.2 in your Postgres source file

🖼️ Public data images#

Do you need a database with data now? We got Sakila, Pagila, Adventure Works or Wide Word Importers, run spawnctl get data-image --public to see the list of public data images and create a data container from those!

📖 Documentation#