microcule + x-ray + docker streaming error

[mikaelkaron]

While trying to implement a microcule powered scraper I've run in to what looks like a broken stream problem, but only if the whole thing is hosted in a docker container (I know, there's a few deps here that could be the root cause but I figured asking for help can't hurt).

To reproduce clone the gist from above:

git clone https://gist.github.com/d10ee3863a6f0773c5386b77abea77fa.git

Build the docker image:

docker build -t microcule-docker

Run the docker image in interactive terminal:

docker run -it -p 3000:3000 microcule-docker

After this hit the following URL's:

http://localhost:3000/request/hook
http://localhost:3000/scrape/hook
http://localhost:3000/scrape/middleware

The second URL breaks in the middle of the JSON. However if you just try a simple npm start and repeat it all runs fine.

One thing to note here is the /request/hook URL works fine in both native/docker so I guess I'm wondering what x-ray does to the stream internally to just close in the middle.

Part of the problem is that debugging this with docker + microcule is a bit tricky and since I suspect that @Marak know a thing or two about hosting hooks inside docker I decided it could be worth filing an issue.

[Marak]

I don't really use Docker at all.

What is the broken stream error?

Is the service timing out? Maybe increase timeout limits?

Either way, if it's working outside docker, it must be a configuration setting related to docker?

[mikaelkaron]

The stream error is just that the output is cut in the middle of the response - nothing in the terminal to show any errors so I'm not sure what's going on inside.

Timeout is already set to maximum.

There's a /scrape/middleware that is the exact same code as in the hook but not executed via spawn and that runs fine both native/docker.

[Marak]

Not sure, try testing more if you can. See if you can replicate the issue without Docker.

[Marak]

@mikaelkaron - Any updates on this?

Are you able to reproduce this with a smaller case?

I'd like to understand why this isn't working in case we need to fix anything in microcule.

Thank you for all your feedback and contributions.

[mikaelkaron]

Not yet. I'm working on an alternative implementation that uses a different scraper and so far so good. It looks like it's something in x-ray or it's does that is the problem in combo with microcule.

Any advice on how one can debug a hook remotely (as we spawn a new process it's a bit hard to track at the moment)