Project

site-to-md

0.0
The project is in a healthy, maintained state
site-to-mdtmaier/site-to-mdHomepageDocumentationSource CodeBug TrackerWiki
A tool that extracts and combines text from HTML files into a single, streamlined markdown document. It provides a command-line interface for easy usage, removes unnecessary HTML elements to reduce token usage, and creates an easily uploadable format for AI tools like Claude AI or ChatGPT. The tool preserves document structure and includes frontmatter metadata.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
 Popularity
Downloads
153
Stars
0
Forks
0
Watchers
1
 Releases
Current version
1.0.5
Total releases
1
First release
2024-12-27
Latest release
2024-12-27
 Pull Requests
Open Pull Requests
0
Closed Pull Requests
0
Merged Pull Requests
5
Pull Request Acceptance Rate
100%
 Development
Primary Language
Ruby
Licenses
MIT
Average date of last 50 commits
2024-12-27
Reverse Dependencies
0

 Dependencies

Runtime

nokogiri
~> 1.18
reverse_markdown
~> 3.0
thor
~> 1.3
 Project Readme

site-to-md

Gem Version Tests MIT License

This command-line tool aggregates content from HTML pages into a single, streamlined markdown file. It removes unnecessary HTML elements to reduce token usage and provides an easily uploadable format for AI tools like Claude AI or ChatGPT.

This tool can also be used to create a llms.txt or llms-full.txt. For this use case, consider to add this tool to the build pipeline of your satic website.

Features

  • Converts all HTML files in a directory (and subdirectories) to markdown
  • Extracts content from <main> tag (falls back to <body> if not found)
  • Preserves document structure with frontmatter metadata
  • Maintains proper markdown formatting for:
    • Headers
    • Lists
    • Tables
    • Code blocks
    • Links
    • And more...
  • Command-line interface for easy integration

Installation

gem install site-to-md

Or add to your Gemfile:

gem 'site-to-md'

Usage

Command Line

Basic usage:

site-to-md convert path/to/site

Specify output file:

site-to-md convert path/to/site -o output.md

Get help:

site-to-md help

Docker Image

You can use the site-to-md tool via a Docker image, making it convenient to include in your build pipeline for static websites.

Example: GitLab CI

Here's an example GitLab CI configuration. This configuration includes a job llms-full-txt that uses the Docker image to convert HTML files in the public folder and generates the llms-full.txt file in the same folder. This

llms-full-txt:
  image: ghcr.io/tmaier/site-to-md:latest
  script:
    - site-to-md convert public -o public/llms-full.txt
  artifacts:
    paths:
      - public/llms-full.txt

Ruby API

require 'site_to_md'

processor = SiteToMd::Processor.new('path/to/site', 'output.md')
processor.process

Output Format

The generated markdown file contains all HTML documents concatenated with frontmatter metadata:

---
path: relative/path/to/file.html
title: Page Title
---

# Content starts here

Document content in markdown format...

================================================================

---

path: another/file.html
title: Another Page

---

More content...

Development

Requirements

  • Ruby 3.2 or higher
  • Bundler

Getting Started

  1. Clone the repository
  2. Open in VSCode with Dev Containers extension installed
  3. Click "Reopen in Container" when prompted

The development container will set up everything you need:

  • Ruby development environment
  • Ruby LSP for code intelligence
  • RuboCop for code style checking
  • Development dependencies

Alternatively, you can run without Dev Containers:

bin/setup

Testing

Run the test suite:

bun/rake test

Run the linter:

bin/rubocop

Dependency Management

We use Dependabot to keep dependencies up to date. Dependabot creates pull requests to update:

  • Ruby gem dependencies (weekly)
  • GitHub Actions (weekly)
  • Dockerfile (weekly)

Release Process

To create a new release, run bin/cut-new-release {major|minor|patch}. Follow semantic versioning guidelines and ensure the CHANGELOG is up to date with the latest changes.

bin/cut-new-release will:

  • Ensure some basic checks pass (e.g., right branch, clean working directory, tests pass)
  • Update the version number
  • Update the CHANGELOG with the new version number and release date
  • Commit the changes and create the new tag
  • Push the changes to GitHub

You need to create a new release manually on GitHub and update it with the content of the CHANGELOG.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/tmaier/site-to-md. This project is intended to be a safe, welcoming space for collaboration.

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Make your changes:
    • Add tests for new functionality
    • Update documentation if needed
    • Ensure tests pass (rake test)
    • Ensure code style checks pass (bundle exec rubocop)
  4. Commit your changes (git commit -am 'Add some feature')
  5. Push to the branch (git push origin my-new-feature)
  6. Create new Pull Request

License

The gem is available as open source under the terms of the MIT License.

About

site-to-md is maintained by maier.io UG (haftungsbeschränkt) and Tobias L. Maier.

Related Projects