PokeFusion/README.md

How to run

DreamGaussion & MVDream

Please install PyTorch & Nvidia CUDA toolkit first.

I run my code on PyTorch 2.1.0 with 11.8 CUDA.

And run these code to build the environment to generate 3D Pokemon models.

git clone https://github.com/dreamgaussian/dreamgaussian.git

cd dreamgaussian

# a modified gaussian splatting (+ depth, alpha rendering)
git clone --recursive https://github.com/ashawkey/diff-gaussian-rasterization
pip install ./diff-gaussian-rasterization

# simple-knn
pip install ./simple-knn

# nvdiffrast
pip install git+https://github.com/NVlabs/nvdiffrast/

# kiuikit
pip install git+https://github.com/ashawkey/kiuikit

# To use MVdream, also install:
pip install git+https://github.com/bytedance/MVDream

Generate 3D Pokemon models

After building the environment.

We provide a script to generate multi type & multi animal type's Pokemon models.

cd ../
python generate.py

It will generate many 3D pokemon models in output/ directory.

And plese run the script to transform the mesh model to cloud points.

pip install -r requirements
python mesh_to_points.py

Camera Calibration

You can find the chess pattern provided by Opencv https://github.com/opencv/opencv/blob/3.4/doc/pattern.png

Please record a video and run calibration scripts

python camera_calibration.py my_video.py

it will save intristic matrix & distortion matrix.

Visualize

Then you can link your webcamera. and run this script to show your Pokemon.

python main.py

Here is ourt DEMO https://youtu.be/nOnE82SKFAM