Sequencescape
Sequencescape is a cloud based and highly extensible LIMS system for use in labs with large numbers
of samples.
- Work order tracking
- Sample and study management
- Capacity management for pipelines
- Accounting
- Accessioning for samples and studies at the EBI ENA/EGA
- Dynamically defined workflows for labs with support for custom processes
- Labware and freezer tracking
- API support for 3rd party applications
Current installation supports over 5 million samples and 1.8 million pieces of labware and is used in
a organisation of 900 people.
Contents
Documentation
GitHub Pages
The Yard documentation is also hosted at GitHub Pages under https://sanger.github.io/sequencescape/.
The documentation is automatically updated via a CI workflow when a merge to master occurs, but you can also trigger it manually against any branch (the branch can be selected using the “Run Workflow” button in the corresponding action).
To preview this documentation, you can spin up a yard server locally using the following command:
yard server --reload sequencescape .
You can then access the Sequencescape documentation through: http://localhost:8808/docs
If the server complains that the stack depth is too deep, this only appears to be a problem when you try to view the documentation without pre-compiling it.
Precompiling is the simple solution and can be achieved with the following.
yard doc
This will pre-fill the cache and allow the server command above to display the documentation without complaining about stack depths.
Sequencescape builds
The documentation is also available within Sequencescape itself, by navigating to the /doc/ route (note the trailing slash). This is automatically included in the CI builds, but can be manually generated for local development with:
bundle exec yard doc --output-dir public/doc
Linting
Yard-Junk is used to check for missing or incorrect documentation. To run the checks:
bundle exec yard-junk --sanity
Requirements
The following tools are required for development:
- ruby (version defined in the
.ruby-version)
- yarn (
brew install yarn)
- node (
brew install node@<version> version defined in the .nvmrc, ensure node is in your PATH)
- mysql client libraries - if you do not want to install mysql server on your machine, consider
using mysql-client:
brew install mysql-client. Alternatively, to install the MySQL required by
Sequencescape (currently 8.4)
Getting started (using Docker)
To set up a local development environment in Docker, you have to build a new Docker image for
Sequencescape. start a stack of services that include a mysql database, and reset
this database contents. You can do all together by running the commands:
docker compose build
RESET_DATABASE=true docker compose up
Or if you are using an Apple M1 Chip:
docker compose build --build-arg CHIPSET=m1
USE_POLLING_FILE_WATCHER=true RESET_DATABASE=true docker compose up
Optionally, if this is not the first time you start the app, you may not want to reset the
database, and you can run this command instead:
docker compose up
With this we should have started Sequencescape server and all required services. You should be
able to access Sequencescape by going to http://localhost:3000 and log in with
username and password admin/admin.
The envvar PRECOMPILE_ASSETS is also available as PRECOMPILE_ASSETS=false docker compose up which will avoid precompiling the assets as Sequencescape is started.
If you are using Apple silicon and encounter any issues, please see Troubleshooting below.
ABOUT LOCAL DEVELOPMENT SETUP You may want to start only the required services for Sequencescape (server and jobs worker) and use your local version of Mysql
instead of the Docker version, in that case you can start this setup with the
command:
docker compose -f docker-compose-dev.yml up
ABOUT RECREATE DOCKER IMAGE If you ever need to recreate the image built on first start (because you made modifications to the Dockerfile file or there have been dependency updates) you can re build the image again with (or see above for M1 Apple chip):
docker compose build
Getting started (using native installation)
This section only applies if you don’t have Docker installed or if you prefer a native installation
of Sequencescape.
Installing ruby
It is strongly recommended that you use a ruby version manager such as RVM or rbenv to manage the
Ruby version you are using. The ruby version required should be found in .ruby-version.
rbenv
If you have the rbenv ruby-build plugin it is as simple as:
rbenv install
It will pick up the version from the .ruby-version file automatically
Automatic Sequencescape setup
To automatically install the required gems, set-up default configuration files, and set up your database run:
bin/setup --skip-server
Manual Sequencescape setup
In the event you have trouble with the automatic process, you may wish to step through the various steps manually.
Installing gems
Bundler is used to install the required gems:
gem install bundler
bundle install
Adjusting config
The config/database.yml file saves the list of databases.
Default setup
-
Create the database tables
bundle exec rake db:setup
-
Install webpacker and the required JS libraries
yarn
Starting rails
bundle exec rails s
Once setup, the default user/password is admin/admin.
Vite
Ensure Node is installed, and in your PATH. You might need to run
bin/vite build --clear --mode=development
Delayed job
For background processing Sequencescape uses delayed_job to ensure that the server is running. It
is strongly recommended to start one for Sequencescape to behave as expected.
bundle exec rake jobs:work
OR
bundle exec ./script/delayed_job start
Message broker
Sequencescape has its own message broker and consumer. To develop this or run it locally, you
must have RabbitMQ installed. It may be easiest to use the docker image https://hub.docker.com/_/rabbitmq.
docker run -d --hostname my-rabbit --name some-rabbit -p 8080:15672 -p 5672:5672 rabbitmq:3-management
It can be useful to follow the rabbitmq logs, to look for broken connections or other problems. To do this using the docker image,
get the container id using docker ps, and then:
docker logs -f <container id>
To start the consumer off listening for messages:
bundle exec warren consumer start
The consumer will run in the foreground, logging to the console. You can stop it with Ctrl-C.
For more warren actions, either use bundle exec warren help or see the
warren documentation
You will also have to change the config in config/warren.yml from type: log to type: broadcast to get
it to actually send messages in development mode.
Credentials
Secrets are managed differently after the Rails 7.2 update. The setup can be done by running the command bundle exec rails credentials:edit locally
This will create a credentials.yml.enc and master.key in the config directory
You can then edit the credentials file with the following command:
VISUAL="nano --wait" bin/rails credentials:edit
These should not be committed to the github repo.
Testing
Rails
Testing is done in one of three ways; using rspec, via rails tests or with cucumber.
-
To run the rspec tests (found in rspec/ dir.):
bundle exec rspec --fail-fast [<path_to_spec>]
-
To run the rails tests (found in tests/ dir.):
bundle exec rake test -f
For a single file:
bundle exec ruby -Itest test/lib/label_printer/print_job_test.rb
-
To run cucumber tests (found in features/ dir.) first ensure you have a sequencescape_test_cuke database configured by running:
RAILS_ENV=cucumber bundle exec rake db:setup
then run cucumber itself:
bundle exec cucumber
For a single file:
bundle exec cucumber features/create_plates.feature
Rspec feature tests with javascript
Where feature tests require javascript, these must be specifically tagged with :js to enable the Capybara javascript driver.
Example:
describe 'Test all the things', :js do
it 'successfully runs javascript' do
# ...
end
end
This can also affect page loading times as well as the parsing of .text results, resulting in changes to capitalisation and whitespace.
Coverage
Local coverage results can be viewed for the last test run by executing:
bundle exec simplecov open
Javascript
-
To run the Javascript Vite tests:
yarn test
For a single file:
yarn test app/frontend/tests/entrypoints/cherrypick_strategies.spec.js:22
Put tests under app/frontend/tests/, mirroring the path of the code they test, and name them *.spec.js.
For example, tests for app/frontend/shared/scanned_barcode.js would go in
app/frontend/tests/shared/scanned_barcode.spec.js. Tests outside app/frontend/ are not run.
Do not put test files in app/frontend/entrypoints/: Vite builds every file there as an entrypoint, so tests
(and Vitest) would be bundled into the production assets.
Rubocop is used for linting and formatting.
bundle exec rubocop --autocorrect
Note that permanent Excludes should be defined in .rubocop.yml, with ‘temporary’ ones
(automatically) listed in .rubocop_todo.yml.
To update .rubocop_todo.yml, execute
rubocop --auto-gen-config --no-exclude-limit
Prettier is used for formatting.
yarn prettier --check .
yarn prettier --write .
(If prettier is not yet installed, run yarn. This should have ben run in bin/setup)
- Prettier rules are configured in .prettierrc.json
- Whole files can be ignored in .prettierignore
- Sections of files can be disabled using #prettier-ignore
Rake tasks
Rake tasks are available for specialised tasks as well as support tasks. Support tasks allow ease
of running standalone scripts multiple times.
A breakdown of the the available tasks and how to run them can be found here
Feature flags
The use of feature flags is encouraged for development.
Flipper is used for flag management and flags can be controlled from the /flipper route.
To create a new feature flag, update config/feature_flags.yml.
Supporting applications
There are a number of services that are needed in certain parts of Sequencescape these are listed
below.
Barcode printing
Barcode printing is carried out by a separate REST service, PrintMyBarcode. The source
for this is also available on GitHub sanger/print_my_barcode
Plate barcode service
Due to DNA plate barcode series being stored in a legacy system in Sanger you are required to use a
webservice for supplying numbers for plates with a simple service.
Data warehousing
There is a client application for building a data warehouse based on the information in
Sequencescape. This is driven asynchronously via RabbitMQ.
See our various clients on GitHub:
sanger/unified_warehouse
sanger/event_warehouse
Schemas
Sequencescape uses RedPanda for schema management. For more information on modifying schemas, see the Schemas Managed By RedPanda documentation.
You can configure Sequencescape to connect to your RedPanda instance by setting the appropriate schema registry URL in the environment or configuration files.
When RedPanda isn’t available, as in development and tests, set
config.sapio_schema_source to local: Sequencescape then reads the schemas
from the files in config/schemas instead. It doesn’t switch by itself: with
redpanda, requests are answered with 503 while the registry can’t be
reached.
Miscellaneous
Lefthook
Lefthook is a git-hook manager that will
ensure staged files are linted before committing.
You can install it either via homebrew brew install Arkweid/lefthook/lefthook or rubygems gem install lefthook
You’ll then need to initialize it for each repository you wish to track lefthook install
Hooks will run automatically on commit, but you can test them with: lefthook run pre-commit
In addition you can also run lefthook run fix to run the auto-fixers on staged files only.
Note that after doing this you will still need to stage the fixes before committing. I’d love to be
able to automate this, but haven’t discovered a solution that maintains the ability to partially
stage a file, and doesn’t involve running the linters directly on files in the .git folder.
Ruby warnings and rake 11
Rake 11 enables ruby warnings by default when running the test suite. These can be disabled with
RUBYOPT='-W0', (eg. RUBYOPT='-W0' bundle exec rake test).
Currently these warnings are excessive, covering both our own code and external dependencies. As it
stands it makes the output of the test suite unusable in travis, as it fills the buffer. These
warnings will need to be fixed, especially in our own code.
NPG - Illumina tracking software
For tracking illumina instruments you need the NPG systems. NPG is linked to Sequencescape via a
cluster formation batch which represents a flowcell.
NPG Software
Troubleshooting
MySQL errors when installing
-
If you are using homebrew with rbenv and run into errors relating to SSL, have a look here
-
If you are upgrading a homebrew MySQL locally and have an error about a missing libmysqlclient dylib file, you may need to redownload the mysql2 gem to fix it i.e. bundle install --redownload
This is because the mysql2 gem is simlinked to the homebrew mysql.
-
If bundle install is failing to install the mysql2 gem, try the below (updating the paths as required):
gem install mysql2 -v '0.5.6' -- \
--with-mysql-lib=/opt/homebrew/Cellar/mysql/ \
--with-mysql-dir=/opt/homebrew/Cellar/mysql/9.0.1_1 \
--with-mysql-config=/opt/homebrew/Cellar/mysql/9.0.1_1/bin/mysql_config \
--with-mysql-include=/opt/homebrew/Cellar/mysql/9.0.1_1/include
MySQL errors after system updates
If you encounter a LoadError at dlopen call to mysql2 bundle because of a missing dynamic library, try the following command to update the cached gem.
gem pristine mysql2
Installing on Apple Silicon (M1)
If installation issues are encountered with Docker on M1 processors, try the fixes below:
-
The docker compose build command fails with any mentions to a processor architecture (‘amd64’, ‘x86’) or the message below:
...
#0 1.528 The following packages have unmet dependencies:
#0 1.568 google-chrome-stable:amd64 : Depends: libasound2:amd64 (>= 1.0.17) but it is not installable
#0 1.568 Depends: libatk-bridge2.0-0:amd64 (>= 2.5.3) but it is not installable
...
#0 1.568 Depends: libxkbcommon0:amd64 (>= 0.5.0) but it is not installable
#0 1.568 Depends: libxrandr2:amd64 but it is not installable
#0 1.581 E: Unable to correct problems, you have held broken packages.
------
failed to solve: process "/bin/bash --login -c apt install -y ./google-chrome-stable_current_amd64.deb" did not complete successfully: exit code: 100
Force docker to use the an AMD image as the base by setting --build-arg CHIPSET=m1 during the build.
It is also recommended to install and enable Rosetta 2.
-
The sequencescape_server container terminates with the error:
Function not implemented - Failed to initialize inotify (Errno::ENOSYS)
Use a polling instead of event file update checker by setting USE_POLLING_FILE_WATCHER=true during the compose up.
[GitHub issue]
Cucumber / RSpec feature chromedriver issues
If you encounter Selenium/Chromedriver issues locally it may be because the latest version of Chrome is yet to be supported by our testing libraries (Capybara). This may require a fix to the Capybara.rb file, e.g. new driver arguments. In the case the fix is not identifiable you made need to rely on a pinned version of Chrome in the CI to run these tests, see Chromedriver issues below for further instructions.
API V2 Authentication
The V2 API has had authentication checks added to it so that other applications calling the API should provide a valid key.
The key is passed by the client application via the X-Sequencescape-Client-Id header.
Keys can be generated via the Rail Console by creating new ApiApplication records and observing the key attribute on them.
As of the time of writing, there are three outcomes to a request made, with respect to API key submission:
- The client calls an API V2 endpoint with a valid API key in the header of the request.
- The response given has a valid status code and the body contains the requested information/confirmation.
- The client calls an API V2 endpoint with an invalid API key in the header of the request.
- The response given has status code 401 Unauthorized and contains a JSON body explaining that a valid API key must be provided for the header.
- The request is logged with the prefix “Request made with invalid API key” including information about the client and the API key used.
- The client calls an API V2 endpoint without the API key header in the request.
- The response is given as if a valid API key was provided.
- The request is logged with the prefix “Request made without an API key” including information about the client.
- The client application should be updated to use a valid API key in future.
Documentation
Example POST requests for all the Sequencscape API v2 resources are available to import into Postman REST Client.
The file Sequencescape API v2.postman_collection.json is stored in the Pipeline Solutions Shared Network Drive (Finder > Go > Connect to Server > smb://files-smb/pipeline_solutions)
Publishing AMQP Messages
Some API endpoints (such as /api/v2/bioscan/export_pool_xp_to_traction) trigger background jobs which are responsible for publishing data to another instance of RabbitMQ.
In the case of the Bioscan Export Pool XP to Traction job, the message goes to the ISG managed RabbitMQ instance.
In order to publish a message, the job must get a schema from a registry.
Under development conditions, you may not have a registry running, hence the default config directs to the UAT instance of PSD’s supported RedPanda.
This means, the first time you publish a message with this schema, you need to be connected to the Sanger network directly or via VPN.
After the first use, a cached file will be created in data/avro_schema_cache so that the registry does not need to be reachable to continue generating messages.
Because this is the first and only job doing this pubishing / RedPanda caching / Avro encoding, etc, there are parts which could be extracted in future if further jobs of this type are created.
This isn’t necessary at this stage, but it seems wise to note the intended pattern of usage here for future work.
CI
The GH actions builds use the Knapsack-pro gem to reduce build time by parallelizing the RSpec and Cucumber tests. There is no need to regenerate the knapsack_rspec_report.json file, Knapsack Pro will dynamically allocate tests to ensure tests finish as close together as possible.
Chromedriver issues
If you encounter CI failures for cucumber and rspec feature tests it may be caused by our CI GitHub runner being on the latest version of Chrome while our testing libraries are yet to support it. To fix this, you can pin the version of Chrome in the CI to an older, known working version. This can be done in the .github/workflows/ruby_test.yml workflow under Setup stable Chrome. If this is required ensure you revert the change once the testing libraries are updated.
ERD
You can create a database entity relationship diagram, by specifying the title and attributes optionally, and view the output:
bundle exec erd
open erd.pdf
The command uses the rails-erd gem.
Updating the table of contents
To update the table of contents after adding things to this README you can use the markdown-toc node module.
To install it, make sure you have installed the dev dependencies from yarn.
To update the table of contents, run:
npx markdown-toc -i README.md --bullets "-"
Copyright © 2007, 2010-2024 Genome Research Ltd.
![depfu[bot]](https://avatars.githubusercontent.com/in/715?v=4)
评论