RPi System/Data Management

Back to RPi Tasks

Summary
Details here: Pi in the Sky - Cloud management?

Basic concept: Enable ways to easily switch and share system images for the R-Pi and manage user data, allowing joint development and easy management of set-ups.
 * Fast/Easy backup of system/data to external storage or network (local or cloud).
 * Management/Repository of base images.
 * Boot from network

This would enable a number of potential scenarios:
 * Keep multiple R-Pi's synced across locations (office vs home or classroom vs home)
 * Manage a multiple users of R-Pi's (i.e. classroom setup, each user's data gets synced with network so each user can continue their own work on any R-Pi)
 * Easy switching of function by booting to multiple images (i.e. Media centre image, programming image, in-car image)
 * Remote management of R-Pi's (by updating the master image)

The project should be able to be divided into a number of smaller projects which when put together may well be able to perform the functions we want.
 * User Data sync/web mount
 * System Image backup/restore (to online/local network/local)
 * Bootloader with boot from online/local network/local image
 * Online image repository

My thinking is that the sooner we can build standard ways to achieve this kind of function, the more likely it is it can become standard for all set-ups and allow a common-ground for community development and sharing (which should allow faster development on the platform overall).

Comments and Ideas
'''Please add any comments and ideas regarding this project here, all are welcome! Or feel free to add anything directly to the rest of the page.'''

Meltwater (20th Dec 2011) - I would love to see this happen, at some level or another. Hopefully those with the skills to implement it will also agree.

iamacarpet (25th Jan 2012) - I'm working on something that might complement this, a cloud based code editor based on Cloud9 IDE that can either be used fully in the cloud or installed locally for schools as I know they like to have most stuff behind their own firewall - A way to set up an environment for kids / people to code en-mass comfortably within about 10 to 20 mins.

I'm all for the hosting of SD card images too - I currently have the capacity and the bandwidth on some of our servers to host atleast 100GB of images for the time being, although I'm not sure I'll be able to let people openly upload them. We'll have to discuss a way to build the archive of these - email me: sam@infitialis.com

peters (2012-06-25) - For labs and classrooms, a classroom-server that provides PXE boot (though PXE boot is not usable yet), git server, credentials server (nis, ldap) and perhaps files server, would do the trick. After boot config could be provided with puppet or cfengine. PXE boot can of course be skipped by preparing sd cards.

User Data sync/web mount
In order to enable seamless switching between physical R-Pi's it would be very useful to Web Sync or Web Mount the user's data so that it will be available to the user regardless of which unit they are using. For circumstances where there is not internet access, Web Syncing (similar to a local dropbox folder) would be very useful, however, in some cases the ability to mount a web drive/ftp location directly will also be very useful (allowing no user data on the local storage).

System Image backup/restore
The ability to perform system image backup/restore (to online/local network/local) will allow users to easily manage their R-Pi. Hopefully, assuming user specific settings and data can be kept separate, this could also allow easy way to produce and build upon community images.

Bootloader
A bootloader with the ability boot from online/local network/local images, will enhance the flexibility of the device. This would allow multiple images to be loaded onto a single card/storage device or multiple standard images to be available on the network, allowing the device to be used for different purposes at different times.

Online image repository
This has to be driven by the R-Pi community, each image will need to be packaged in an easy to use format, with clear documentation about it's contents and use. Ideally, the bootloader/OS could link into such a repository so that updates and releases can easily be used.