Dgraph and CI CD - Bitbucket pipelines

Users
1

In development I use DGraph with docker-compose as per your docs. I have a dev and test setup, its been working great.

I am currently struggling to setup a CI CD pipeline in Bitbucket. The suggested solution for Bitbucket, for other databases is to define services which are themselves docker images.

See docs here:

It does not appear that you can call commands on the docker images you define as services. (docs are a bit light).

In this scenario however each docker image just runs and exposes a single database automagically. As Dgraph requires multiple docker images to setup I cannot just use the dgraph/dgraph image.

I have been looking into trying to run docker in docker with compose, and creating my own single dgraph image (and failing). Ideally I would use my local test compose file as any base for my CI CD integration tests to keep them in sync. Docker in Docker does however seem to be discouraged Using Docker-in-Docker for your CI or testing environment? Think twice.

I am a bit lost as to how to get this sorted, and the best approach, any help would be appreciated. I figured I should reach out here as I assume you guys have some CI CD setup happening. Maybe you would be open to having an official single docker image for this purpose that just loads Dgraph (Zero and Ratel) on an exposed port with no config.

My test file for docker compose for reference.

version: "3.2"
services:
  zero-test:
    image: dgraph/dgraph:v1.0.3
    volumes:
      - type: volume
        source: dgraph-test
        target: /dgraph-test
        volume:
          nocopy: true
    ports:
      - 6081:6081
      - 5081:5081
    restart: on-failure
    command: dgraph zero --port_offset 1 --my=zero-test:5081
  server-test:
    image: dgraph/dgraph:v1.0.3
    volumes:
      - type: volume
        source: dgraph-test
        target: /dgraph-test
        volume:
          nocopy: true
    ports:
      - 8081:8081
      - 9081:9081
    restart: on-failure
    command: dgraph server --my=server-test:7081 --memory_mb=2048 --zero=zero-test:5081 -o 1

volumes:
  dgraph-test:
2

Did you mean Zero and Server? Why do you require Ratel on a CI/CD system?

One solution that comes to mind is that you can run both Zero and Server in the same container and then wait for the cluster to be healthy (by checking /state endpoint on Zero and making sure a leader has been elected for group 1) using some script. Maybe give that a try and if it doesn’t work I can try it.

3

@pawan

Thanks for the reply.

I did get it working, but your one container solution might be better. I am not sure how best to set this up. The docker documentation warns you against using more than one service per container, so I hesitated going down that path. However in this case it might be the best bet.

I ended up creating my own Docker images (2, 1xServer, 1xZero) to get it working, based off your docker image. They are very simple but allow me to just add them both as services (Bitbucket pipeline terminology) to the bitbucket pipeline and have everything *just work.

Even though I have it working I am not sure it’s the best implementation and I feel it might be worthwhile for Dgraph to have an official solution for this. I think it will be a common need for build pipelines other than bitbucket. A lot of other CI/CD servers also allow you to hook in a test db from a docker image.

Just FYI the main issue is that other docker databases that Bitbucket references in its pipeline feature all just auto startup on a port, and that’s the core of the issue I had, that the Dgraph docker image cannot just auto startup. I know that your DB is a bit different because it’s distributed, but it’s just something for you to think about. Links to the other DB docker files from the Bitbucket pipelines documentation here: https://confluence.atlassian.com/bitbucket/test-with-databases-in-bitbucket-pipelines-856697462.html

My solution

Create a zero docker image that auto instantiates itself on localhost 5081

FROM dgraph/dgraph:v1.0.3
EXPOSE 5081 6081
CMD dgraph zero --port_offset 1 --my=localhost:5081

Create a server docker image that auto instantiates itself and connects to zero on localhost

FROM dgraph/dgraph:v1.0.3
EXPOSE 8081 9081
CMD dgraph server --memory_mb=1024 --zero=localhost:5081 -o 1

I build an publish both of these to docker hub.

I then setup my build pipeline to have both images as services. By default bitbucket merges all of the networks so every docker image can access any other on localhost:$EXPOSED_PORT.

Bitbucket pipelines config

image: weboaks/node-karma-protractor-chrome
pipelines:
  default:
    - step:
        script:
          - mv .npmrc_config .npmrc
          - yarn
          - yarn run ci-test-frontend
          - yarn run test-db-up
          - yarn run test-backend
        services:
          - docker-zero
          - docker-server
definitions:
  services:
    docker-zero:
      image: cameronbatt/dgraph-zero
      memory: 1024
    docker-server:
      image: cameronbatt/dgraph-server
      memory: 1024
1 Like
4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.