Creating SWOOD materials from manufacturer site with bit of programming skills

Hi everyone, I have been doing some hobby work in SWOOD and I wanted to have a list of materials created from the manufacturers website. We can use a bit of web scrapping to get the data we need.

For my case I wanted to get Kaindl boards data from https://www.kaindl.com/en/boards/

To do that we use some standard node packages:

• fs and path from node standard library
• cheerio to do the html parsing
• axios to fetch page html and images
• csv to do csv generation

Here’s the whole code with NOTEs inside to explain the parts.

const fs = require('fs');
const path = require('path');
const cheerio = require('cheerio');
const axios = require('axios');
const stringify = require('csv-stringify/lib/sync');

async function main() {
    // NOTE: we can switch comment on next two lines if you want to use the page from file system instead of loading it from web every run
    // const page = fs.readFileSync('page.html').toString();
    const page = (await axios.get('https://www.kaindl.com/en/boards/')).data;
    const $ = cheerio.load(page);

    // NOTE: cheerio uses css selectors to target the page elements
    const styles = $('p.ft4.f-bold.color-dark.product-style-name');
    for (let i = 0; i < styles.length; i++) {
        // NOTE: I wanted to have one csv per file.
        // NOTE: If you want all styles in one csv file, move the line below and lines saving the csv - outside the loop
        const csv = []; // create CSV per file

        // NOTE: Getting the style data
        const style = styles.get(i);
        const $style = $(style);
        const styleName = $style.text();
        console.log(`Working on ${styleName} style`);

        // NOTE: Getting products inside current style
        const styleProducts = $style.next().find('.pattern-grid__item.js-product-pattern');
        for (let styleProductIndex = 0; styleProductIndex < styleProducts.length; styleProductIndex++) {
            const product = styleProducts.get(styleProductIndex);
            const $product = $(product);

            // NOTE: Getting the product data. If you want, you can use additional files to map data that is missing.
            // NOTE: For example, you can use price list to match the name of panel and properly set the cost
            // NOTE: or to enter proper density for certain styles
            const filterData = JSON.parse($product.attr('data-filter'));
            const name = $product.find('.pattern-grid__item-info > .js-product-pattern-code').text().trim();
            const thickness = parseInt(filterData.th[0]);
            const grain = styleName.toLowerCase().startsWith('wood') ? 1 : 0;
            const cost = 0;
            const density = 654; // https://www.kaindl.com/fileadmin/user_upload/Produktinformationen/Downloads/EPD_Particle_board__raw_and_coated_E.pdf
            const colorRGB = ''; // r,g,b
            const imageUrl = $product.find('.image-wrapper > picture > img').attr('data-src');
            // NOTE: Change to suit your needs
            const imageName = `Keindl ${styleName} ${name}.jpg`; // Need to be in one folder
            
            const imageLocalPath = path.join(__dirname, '..', imageName).toString();
            if (!fs.existsSync(imageLocalPath)) {
                const imageRequest = await axios.get('https://www.kaindl.com/' + imageUrl, {
                    responseType: 'arraybuffer'
                });
                fs.writeFileSync(imageLocalPath, imageRequest.data);
            }
            const textureDirection = 1; // 0 None, 1 Horizontal, 2 Vertical
            const useInSawCut = 0; // 0 disabled, 1 enabled
            const sawCutReference = name;
            const feedGrainCorrectionAngle = 0; // number
            const feedGrainCorrectionFactor = 0; // number
            // NOTE: Change to suit your needs
            const folder = 'Keindl\\'  + styleName; // Folder to put the material
            const materialType = 0; // 0 panel, 1 laminate, 2 compound

            csv.push([name, name, thickness, grain, cost, density, colorRGB, imageName, textureDirection, useInSawCut, sawCutReference, feedGrainCorrectionAngle, feedGrainCorrectionFactor, folder, materialType]);
        }
        fs.writeFileSync(styleName + '.csv', stringify(csv, {
            delimiter: ';'
        }));
    }
}

main();

If you want to test this code, copy it in file code.js , navigate with terminal/console/cmd to containing folder and run npm i axios cheerio csv to install required packages. After packages are installed run node code.js . First time it will take a bit to create the images but after it will be very fast. After program runs successfully, containing folder will contain a bunch of .csv files that can be imported into SWOOD materials library.