All systems interact with external components at some point. This can be a database, another microservice from your own team, a microservice from another team, or even a service outside of your company. Most interactions with databases can be tested by spinning up a database testcontainer. Other microservices maintained by your own team are also relatively easy. You’re well aware of how they’re deployed, how they behave and what dependencies they have. In most cases you can deploy these in a generic testcontainer. This leaves us with external services, these are usually the most challenging. There are a few common approaches here:
The first option is preferred, but not always available. The next best option would be to deploy the service in a generic testcontainer. This is assuming you have access to the docker image of this service, which isn’t always the case. Also, the service might depend on other services too. Trying to deploy all this can quickly become too complicated.
The most common approach is to mock the external service. But there’s one major problem with this: the behavior of your mock might not match the actual behavior of the external service. Also, if you decide to use a newer api version, it’s highly unlikely that you check if the behavior of your mocks still matches reality. And finally, it’s just a lot of work. Manually writing the expected output for each interaction becomes tiring real quick.
So what’s the alternative?
Stoplight Prism is a tool to generate a mock API based on an OpenAPI specification. I prefer to also run this in container, so that my test can spin up the prism service by itself. Let’s take a look how this would work in a simple python example:
post_example.py
import requests
def add_item(url: str, item: str) -> bool:
path = "/api/items"
headers = {"Content-Type": "application/json"}
data = {"name": item}
response = requests.post(f"{url}{path}",
headers=headers,
json=data)
return response.status_code == 200
The method we want to test (add_item(url, item)) makes a POST request to an external service. We want to test that our interaction with this external service is correct. Is the URL correct? Is the body correct? Do we correctly make the request? We could do this by writing a mock, but as mentioned before, our mocks might be wrong. Let’s see how we can use the provided openAPI specification to create a mock service:
resources/api.yaml
openapi: 3.0.0
info:
title: Items API
version: 1.0.0
paths:
/api/items:
post:
summary: Add a new item
requestBody:
required: true
content:
application/json:
schema:
type: object
properties:
name:
type: string
responses:
'200':
description: Successful operation
test_post_example.py
import pytest
import os
from testcontainers.core.container import DockerContainer
from testcontainers.core.waiting_utils import wait_for_logs
class TestService:
@pytest.fixture(scope="module", autouse=True)
def service_container(self, request):
prism = DockerContainer("stoplight/prism:4", init=True)
prism.with_volume_mapping(os.getcwd() + '/resources', '/apis') \
.with_exposed_ports(4010) \
.with_command("mock -v 'debug' -h 0.0.0.0 /apis/api.yaml") \
.start()
wait_for_logs(prism, "Prism is listening on http://0.0.0.0:4010")
def remove_container():
prism.stop()
request.addfinalizer(remove_container)
return prism
def test_add_item(self, service_container):
url = "http://localhost:" + str(service_container.get_exposed_port(4010))
assert add_item(url, 'test') == True
The test itself is defined in test_add_item(self, service_container). Before this method is called, a Prism container spins up. The OpenAPI yaml configuration is passed to the container by attaching it in a volume. We wait until the container is ready (by waiting for logs), and then continue with the actual test.
Now when we call add_item() in the test, it will send the request to the Prism container, which validates the validity of the request. If our request doesn’t match the API specification, we’ll know about it!
The way our test is set up now gives us no control over the response of Prism. This might be enough in some cases, but sooner or later you’ll want to test some logic that depends on the values returned by the external service. Let’s take a look at what that might look like in practise. Let’s consider an API that returns books based on their title. The response includes a ‘status’ field that can be “IN STOCK” or “SOLD OUT”. We use this API in a function that returns ‘True’ if the book is in stock, and ‘False’ if the book is Sold out.
Here’s what our method looks like:
import requests
def in_stock(url: str, book_name: str, headers: dict) -> bool:
response = requests.get(url + '/api/books',
params={'name': book_name},
headers=headers)
data = response.json()
return data['status'] == 'IN STOCK'
We cannot make any assumptions as to what our Prism mock server will return. Therefore we cannot guarantee that this method will return True or False when we test it. We need a way to tell Prism what to respond to a certain request. To make this work, we need to add predefined examples to the OpenAPI specification. Prism allows us to define which example to return depending on a value passed through the header. Notice that out function definition now allows us to pass headers to the request.
openapi: 3.0.0
info:
title: Bookstore API
version: 1.0.0
paths:
/api/books:
get:
parameters:
- name: name
in: query
required: true
schema:
type: string
responses:
'200':
description: OK
content:
application/json:
schema:
type: object
properties:
name:
type: string
status:
type: string
enum:
- IN STOCK
- SOLD OUT
examples:
inStockExample:
value:
name: "The Great Gatsby"
status: "IN STOCK"
summary: Example of a book in stock
soldOutExample:
value:
name: "1984"
status: "SOLD OUT"
summary: Example of a sold out book
class TestService:
@pytest.fixture(scope="module", autouse=True)
def service_container(self, request):
# omitted, see previous example
def test_in_stock(self, service_container):
url = "http://localhost:" + str(service_container.get_exposed_port(4010))
assert True == in_stock(url, 'The Great Gatsby',
{"prefer": "example=inStockExample"})
def test_sold_out(self, service_container):
url = "http://localhost:" + str(service_container.get_exposed_port(4010))
assert False == in_stock(url, '1984',
{"prefer": "example=soldOutExample"})
The downside here is that we need to modify our production code so that we can inject headers in the test. This is not always desired, and sometimes it’s not even possible. The OpenAPI specification also needs to include examples that we can use in our tests. In most cases you’ll have to add these manually. So if the external service updates their API, you cannot just swap the specification and be done with it.
What if we don’t want or can’t inject headers into our HTTP request? Stoplight has an example of how to implement a proxy server to work around this issue in this repository. It’s a service that sits between your test and the Prism mock server. This proxy looks at your request and adds the right ‘prefer’ header, based on predefined logic. It then forwards this query to Prism, and returns the result as-is.
Adding the examples to the OpenAPI spec seems a bit more problematic. It’s not ideal to modify the specification directly, because then we cannot easily update it. At the time of writing, I haven’t found a nice solution to add examples to OpenAPI specifications. It’s not that difficult to write your own, but it comes with the added problem that Prism doesn’t validate examples to the spec.
It’s perfectly possible to create an example that doesn’t match the defined response format. Currently I use Prism alongside traditional mocks. I use Prism to verify that my interaction with the API matches the API specification, and I use mocks to test my logic based on different potential responses.
]]>Git is a set it and forget it setup for most people. Every time I get my hands on a new machine, I always have to go through the same guides. It’s not difficult, but since I only do this every year or two, it’s not something I tend to remember. Recently I configured git with multiple accounts (private and work), which is a bit more challenging than just blindly following a guide on how to set up an ssh key.
In this post I’ll try to explain step by step how to set up git with two accounts. How to commit with different usernames, how to use the right ssh key depending on which repository you’re pushing to, and how to sign your commits with different GPG keys.
In this example I assume two github accounts: one private and one work account. On the private repository, I want to make commits with my private email address. On The work repository, I want to make commits with my company email address.
I assume you’ve already set up git with one account. When you run git config user.name && git config user.email, it should show your git username and email address. This user data is added to all your commits.
One way to use a different configuration on different repositories is by setting git config --local user.name "My Name". However, the annoying thing is that you have to remember to do this for every new repository. Luckily git introduced conditional configuration includes in v2.13. In my case, I clone private repositories to ~/Documents/private/repos, while work repositories go to ~/Documents/work/repos. To use a different user name and email based on the repository location, I modify ~/.gitconfig as follows:
[includeIf "gitdir:~/Documents/private/repos"]
path = .gitconfig-private
[includeIf "gitdir:~/Documents/work/repos"]
path = .gitconfig-work
To set up the actual username and email, I need to create two new files: ~/.gitconfig-work and ~/.gitconfig-private. These look like normal git config files. For example, gitconfig-work might look like this:
[user]
name = my name
email = name@company.com
To test if this works, cd to any repository and run git config user.email. It should return your private email address on private repositories, and your work email for work repositories.
Now that we can git commit under different usernames, let’s see how we can git push using different credentials. To do this we first need to create a new ssh key and add it to our github account. Make sure to give the new key a different name when prompted. I name both keys ~/.ssh/id_ed25519_gh_private and ~/.ssh/id_ed25519_gh_work.
ssh-keygen -t ed25519 -C "name@company.com"
Then navigate to your account settings on github and click on ‘SSH and GPG keys’. Copy the public key (the file ending in .pub in the ~/.ssh/ directory) and add it as a new SSH Key on github. Make sure to log in to the right github account first.
The only thing left to do now is configure git to use the right ssh key on the right repository. Remember the two .gitconfig files we created earlier? We’ll add an extra section to both files to configure the ssh command. My ~/.gitconfig-work file now looks like this:
[user]
name = my name
email = name@company.com
[core]
sshCommand = ssh -i ~/.ssh/id_ed25519_gh_work
If everything is set up correctly, you should now be able to push and pull from both private and work repositories.
The process of initially setting up a GPG key can be confusing, but configuring multiple keys is relatively easy. There’s not really any configuration required. If your commit is made with a certain name and email address, then the right key will automatically be used to sign the commit.
As a quick reminder, setting up a new gpg key with github can be done as follows:
# First generate the key
gpg --full-generate-key
# find the right key id
gpg --list-secret-keys --keyid-format=long
# Find the right id in the output after the algorithm:
# sec rsa4096/546E539071567BD2 2024-03-06 [SC] [expires: 2028-03-06]
# the key here is '546E539071567BD2'
# export the public key
gpg --armor --export 546E539071567BD2
From this output, copy the part beginning with -----BEGIN PGP PUBLIC KEY BLOCK----- and ending with -----END PGP PUBLIC KEY BLOCK-----. This key should then be added under ‘GPG keys’ on github.
If you want to add GPG signing to only one account and not for the ohter, then this can also be done by modifying the matching ~/.gitconfig files. For example:
[commit]
gpgsign = true / false
That’s it! Now git is fully set up to work with multiple accounts.
]]>Unlike statically typed languages like Java, C++ and others, Python is a dynamically typed language. This means that the type of the variable isn’t fixed. To illustrate this, let’s look at some valid Python code in the following example.
def addOne(number) :
return number + 1
print(addOne(5)) # prints '6'
print(addOne("5")) # TypeError: can only concatenate str (not "int") to str
So while it does throw an error when the code is executed, it won’t prevent you from packaging and deploying your code. However, the same thing is not possible in a statically typed language like java:
int addOne(int num) {
return num + 1;
}
void main() {
System.out.println(addOne(1));
System.out.println(addOne("1"));
}
// error: error: incompatible types: String cannot be converted to int
This is a compile error, so even if we’d want to, we cannot run this code. The compiler doesn’t let us. In many cases, this is an advantage. It’s impossible to accidentally pass a number to a function that expects a string. Also, the function definitions and how to use them tend to be more clear. Especially when codebases grow larger, or when dealing with external libraries, it isn’t always obvious which types are used where if it isn’t explicitly defined.
So how does python deal with this?
By default, python does not require you to define the type when declaring a variable. However, Python 3 introduced type hinting in the typing module. While this is great, it’s actually nothing more than adding documentation to your code according to a predefined format. Let’s look at the previous example again, but this time using type hints:
def addOne(number : int) -> int:
return number + 1
print(addOne(5)) # prints '6'
print(addOne("5")) # TypeError: can only concatenate str (not "int") to str
The types of the variables are defined here, as well as the type of the return value. However, it didn’t prevent anything. In fact, you can claim that the function takes in a String and returns a list. The code will still run, and the error will still be the same.
def addOne(number : str) -> list:
return number + 1
All these type hints are nothing but documentation, that’s kept up to date by nothing more than good intentions. Anyone who has worked on large codebases know that documentation quickly becomes outdated, unless there’s a strong culture of doing code reviews, or some kind of an automated check is in place, to match the documentation to the actual code.
The only way to get true benefit from type hints, and to keep them up to date, is to introduce a linter that checks for them. A linter is a tool that performs static analysis on your code. In other words, it checks for errors without actually running the code. The first python linter that does type checking is mypy. Let’s take a look what it outputs for our previous example:
mypy example.py
example.py:5: error: Argument 1 to "addOne" has incompatible type "str"; expected "int" [arg-type]
Exactly what we need! We can now add this linter to our CI/CD pipeline, and stop typing issues before they’re even deployed. In conclusion, type hints are nice, but without a linter to enforce their correctness, a lot of their value is lost.
]]>You might have already heard that using the :latest tag in docker is bad practise. This is true for several reasons. For starters, a new image might be pushed without adding the latest tag. Meaning that if you pull image:latest, it might not actually be the latest image.
On top of that, if you’re running a new container, you might not really be using the latest image, even if you’re using the latest tag. Not all tools will automatically check if a new version is available. If you already have an image tagged as latest locally, docker will not automatically pull a new image.
I’d like to extend this reasoning for other tags. Let’s say you already pulled image:0.0.1, but someone else pushed a new image using the same tag. Next time you execute docker run image:0.0.1, it will not automatically pull the updated image. You’ll still be using the ‘old’ version 0.0.1.
This problem can be avoided by using docker run --pull=always, but what about containers that are already running? If you’re not careful, you will have two different versions of version 0.0.1 running. If that sentence sounds confusing, that’s because it is.
Don’t overwrite docker tags.
I wouldn’t be writing this if I hadn’t recently run into this problem. Our container registry allowed to overwrite docker images, and our continuous integration pipelines (CI/CD) did not automatically bump versions. This was a project with very little changes once it was set up, and any change usually was accompanied with a redeployment of most of our infrastructure.
This was fine for the first couple of months, but then Dependabot was introduced on here, as well as on some other projects. It started occasionally opening pull requests, which in turn automatically triggered a run on our CI/CD.
I already talked about bad practise: allowing overwriting docker images, but that in itself would not have caused problems. There was a bug in our CI/CD, that also published docker images (under the same tag) when running checks on Pull Requests (PR). Since there were so few changes, and all changes were usually immediately accepted, this remained unnoticed for a long time.
Dependabot automatically updated some dependencies, this triggered an automated build of the project, which was automatically published on an existing tag.
The final piece of the puzzle was Kubernetes, where our application was running. Kubernetes will not automatically redeploy a new image when it’s available, but it will always pull the image when starting a new container. Several days after the PR from Dependabot was opened, Kubernetes had to restart a failed pod, and pulled the new image.
At this point we had two different instances of the same application running side by side, but looking at the pod definition, they were identical. This whole process did not involve any human interaction.
While this could have been avoided by not publishing builds when running CI/CD on pull requests, the real issue was pushing images with tags that already existed.
Unfortunately Azure Container Registry (ACR), where our images are hosted, does not allow automatically locking image tags once in use, but you can lock existing tags individually:
az acr repository update \
--name myregistry --repository myrepo \
--write-enabled false
It can be a good idea to run this as part of your CI/CD each time an image is published. With some registries, such as JFrog Artifactory you can prevent overwriting existing deployments.
]]>As with all things in life, nothing is as easy as it appears. Let me walk you through a bug I dealt with a while ago, and share why it could ultimately be blamed on ‘smart’ DTO’s.
I had just started working on a new project. As usual, I first made sure it can be built and tested locally. Almost immediately it became obvious that even the unit tests in this project behaved rather unpredictably. Some tests would sometimes fail and sometimes pass, even without changing anything. Surprisingly not everyone on the team was suffering from this problem. Tests would also never fail when they ran in isolation. There was only an issue when I ran the entire testsuite.
But who ever runs the whole testsuite locally? Everyone just pushes their code and let the CI/CD pipeline do the rest. Surprisingly though, when running the whole testsuite on Jenkins, it behaved as expected. So the problem remained largely unnoticed.
All good and well, but what does this have to do with DTO’s? I’ll get to that, don’t worry. It will quickly make sense once we take a closer look at what one of the unstable tests looked like:
@Test
void createsNewObject() throws Exception {
final MockHttpServletRequestBuilder request =
RestDocumentationRequestBuilders.put("/endpoint")
.content(asJsonString(MY_DTO_OBJECT))
.contentType(APPLICATION_JSON);
when(processor.process(MY_DTO_OBJECT)).thenReturn(NEW_OBJECT_RESPONSE);
final ResultActions result = mockMvc.perform(request);
result.andExpect(status().is(Matchers.in(new Integer[]{200, 201})));
}
This test failed with a NullPointerException, somewhere in the controller we’re testing. More specifically, processor.process() returned null. Which is weird, since the argument is in fact MY_DTO_OBJECT, and we explicitly state that in this case, it should return NEW_OBJECT_RESPONSE.
The attentive reader probably already figured out that this cannot possibly be correct. First of all, because the test wouldn’t fail if it was, and secondly, because we never directly pass MY_DTO_OBJECT to the controller. It’s passed as a JSON object and rebuilt by Spring. Therefore, the argument cannot be equal.
But that shouldn’t matter. Mockito argument matching calls Object.equals(Object other); And we do in fact Override the equals method on our DTO. Let’s take a closer look at what kind of DTO we’re dealing with. It’s heavily simplified though, just to illustrate the point.
public class MyDto {
private final URL theURL;
public URL getTheURL() {
return this.theURL;
}
public boolean equals(final Object o) {
if (o == this) return true;
if (!(o instanceof MyDto)) return false;
return Objects.equals(this.getTheURL(), ((MyDto)o).getTheURL());
}
public int hashCode() { /* skipped */ }
}
A URL looks pretty innocent, but a quick Google search shows that URL.equals() is completely broken. In fact, this has been known since 2006. Mystery solved! Our argument matching did not work since the URLs in our DTO need to be equal, and that can behave unpredictably because it depends on your DNS settings.
Most discussions on this problem will tell you to use URI instead, but I’d go a step further and say that for DTO’s, even URI is overkill. The Data Transfer Object should not know how to use this data, it should only transfer the data itself. Let the end-user deal with creating an URI from a given String, and handle all the exceptions that go with it.
For DTO’s, POJO‘s suffice.
]]>