artin/nzbhydra2
1
0
Fork 0
mirror of https://github.com/theotherp/nzbhydra2.git synced 2026-02-06 11:17:18 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Update to Spring Boot 3, new h2, use graalvm, GH actions (#816)

Browse Source 
* Handle database recreation errors better

* Handle different db usernames

* Improve shutdown

* Don't run startup problem detection in main thread

* Fix stats calculation with h2 2

* Fix unit tests

* Disable some tests not yet fixed (if ever)

* Use SCRIPT TO to backup database as it verifies integrity

* Only delete backups if a newer successful one exists

* Fix error while analysing log file for OOM errors

* Fix CSRF

* Log Spring stuff on INFO

* Add features for v5 to changelog

* Don't try to remove windows tray icon cause it results in a hang

* Fix notification history pagination

* Fix shutdown behavior, prevent misleading error messages

* Remove javax.annotation and .activation

* Fix baseConfig.yml

* First step to native build

Compile runs, executable can be started, but has errors. More work needed.

* First step to native build

RUns under windows, HtmlUnit error, great progress!

* Instantiate indexers and downloaders directly to make it work with native

BeanFactory.createBean and .autowireBean do not work

* Disable javascript in OpenPortChecker to fix charset error in native

* Test Github Actions

* Test Github Actions

* Test Github Actions

* Test Github Actions

* Adapt wrappers to support native images

* Update wrapper build to use docker containers

clean folder structure
Use v2 binaries for native support

* Add build scripts, remove circleCi

* Native build github actions

* Don't write yaml, use in-memory db in native build mode

* Upload native image artifact

* Test self-hosted runner

* Build on all three OSes

* Build on all three OSes

* Build on all three OSes

* Build macos

* Build windows

* Build windows and macos

* user proper paths for uploading artifacts

* user proper paths for uploading artifacts

* user proper paths for uploading artifacts

* All three native images are built and uploaded

But linux native not runnable under WSL, trying ubuntu 20.04

* Build static binary for linux

* Make wrapper and update manager support both 4.x and 5.x, native and generic

* Try building static image again

* Commit wrapper binaries to LFS

* Move files back to include folders

* Adapt build-and-release.cmd to shell script

* Prepare release via actions, add generic release

* Prepare release via actions, add generic release

* Update other package to 3.0.0

* Delete old docker files

* Support dry run in release plugin and github action

* Always use github token in release if available

* Fix PhantomJS compile error, disable actions on push

* release needs build

* Show file structure on runner

* Build on self-hosted windows runner

* Disable graalvm watchdog so hopefully the windows build completes

* Build on push, update dependencies

* Build on windows

* Determine release type from existance of files instead of json

* Check version of executables before releasing

* Scripts for test docker containers

* Show system tray via wrapper, make main process headless

* Native hints

* Migrate JUnit 4 to 5

* Fix auth error when no internal api key was provided

* Move integration tests

* First step to system tests

* Run system tests on github actions

* Run system tests on github actions

* Delete tests using old mockwebserver, use mockserver in system test

* Delete tests using old mockwebserver, use mockserver in system test

* Disable native image build to get to working systemtest faster

* Specifically download coreLinux

* Upload and download docker images

* Save images in separate step

* Fix docker port mapping syntax, keep docker images for one day only

* Fix docker daemon syntax

* Check out repo before running tests

* Move config validation to separate classes

* Start moving config classes to separate module

* Finish moving config classes to separate module

* Initialize test instance with sensible data

* Push and use docker images using ghcr

* Use custom health check for hydra docker container

* Disable CSRF, use debug level for log file

* Tests for debug infos

* Use  ${{ github.workspace }}

* Use temp folder for hydradocker data

* Write simple downloader web tests

* Use /tmp as data folder

* Test download of zipped NZBs

* Build shared module before running tests

* Load joptsimple resource bundle

* Improve error page

* Log spring security on debug level

* Build new native image

* Use new image

* USe new reflect-config.json. Do I have to do this every time?

* Reenable native build

* Add reflection marker for native hints

* Ensure all relevant classes have a reflection marker

* Call TMDB api directly instead of using library

* Use correct URL in mocked newznab results when in pipeline

* Fix session error in IndexerUniquenessScoreSaver

* Clean up code after migration to Java 17

* Backup tests

* Remove spring profile from core dockerfile so that it can hopefully be loaded from env

* GOOD STATE: Build native before doing tests

Image is built, Two tests fail, but most succeed, yay

* Test configuration of external tools

* Run unit tests, upload test reports

* Fix test being OS dependent (oops)

* Fix test being OS dependent (oops)

* Fix another OS dependent test, try other test report action

* Create system test report

* Try controlling native build by using "skipnativebuild" in the commit message

* Controlling native build by using "skipnativebuild" in the commit message works, yay

* Prepare entity TOs

* Merge shared config and mapping modules

* Merge shared entities and mapping modules

* First simple test for history

* Clean install for native image build

* Commit missing files

* More tests for search history

* Disable repo.spring.io because central repos should be preferred

* Test download history

* Remove repo.spring.io repo

* Upload settings.xml for debugging

* Try custom settings.xml to prevent using repo.spring.io

skipnativebuild

* Remove settings.xml stuff, try running *arr dockers directly

skipnativebuild

* Fix sonar host

* Don't skip prepareArtifacts as other jobs depend on it

* Run all containers manually

skipnativebuild

* Set spring profiles in test env again

skipnativebuild

* SEMI_GOOD_STEP USe custom docker network in system test

Communication between containers works, still bytecode error and BackupData#getBackupFile not working

skipnativebuild

* Remove lazy loading in SearchResultEntity, use TransactionTemplate in IndexerUniquenessScoreSaver.java

* Remove lazy loading in IndexerApiAccessEntity.java

* Make entities final to perhaps prevent lazy loading?

* Remove ToStringSerializer for entity IDs

* Support TVMAZE ID in search history

* Remove names from docker run commands

* Try using docker host network

* Try systemtest network again

skipnativebuild

* Print build version and timestamp at startup

* GOOD_STEP Run mockserver first so core can access it

Only two failing tests remain

* Print version and timestamp even if in docker

* Use mapped volume for black hole

* YESSSS GREEN PIPELINE Use a category for *arr which is supported by a configured indexer

* Add stats test, is kinda flakey

* Remove integration tests

* Install java in docker container for db migration

* First step to migration test

* Remove migration module from mvn call

* Generate custom assertions for mapping module

* Add caps check test

* Add media info test

* Add news test

* Add notifications test

* Reenable migration test, perhaps it works now?

* Add missing assertion files

* Try different docker sock, add unmarshaller resource bundle

* Don't trim stacktraces in maven, log unparsable indexer output

* Use hopefully correct migration tmp folder, create it before using

* Use different md to html renderer

* Make caps inherit XML

* Fix news test, create /tmp/data/v1MigrationDataFolder in action

* Try different temp folder

skipnativebuild

* Print version earlier

* Run migration docker directly

* Print version differently

* Wait for healthy containers

* Print version and timestamp from properties

* Wait for healthy containers

* Use latest docker image when skipping nativebuild

skipnativebuild

* Give containers a bit of time

skipnativebuild

* Use docker-compose

skipnativebuild

* Try java from path if executable not found

* Try to get more current artifact

* ALL GREEN YAAAAAAY Build image in first job if configured, then just use latest in system test

* Precheck that wrapper executables are newer than source files

* Run all for core and migration, respectively

* Remove discord bot, publish via maven plugin when releasing

* Use container name in external hydra url

* Move database migration to spring component

* Use SLF4J in system tests

skipnativebuild

* Use 5076 on container

skipnativebuild

* ONE LEFT Wait a bit between file mod date changes

Only blackhole test not working anymore

* Use name of test component (core, v1Migration) in black hole folder

* Log properties and docker mounts

skipnativebuild

* ALL GREEN WITH two test runs Mount blackhole folder for core

skipnativebuild

* Check changes files to decide if native image is built

* Check changes files to decide if mock server is built

* Send build decision messages to notice

* Use changelog.yaml instead of .json

* Fix startup of wrapper without internal api key

* Use new wrapperHashes2.json

* Write messages for discord and wiki

* Ensure database integrity before creating backup

* Show warning to user if automatic backup has failed

* Show better error message when user triggered backup creation failed

* Shoe proper error message when debug infos creation / download failed

* Update wrapper hashes

* Add loadtest, cache OKHTTP clients

* Improve performance of search result parsing by getting rid of regex

* Compress native image binary

* Update release script

* Update release script

* Check for environment variables in release script

* Check for untracked files or uncommitted changes in release script

* git ignore token files

* Read tokens from files

* Color output for release script

* Remove call from release script (only needed on windows)

* Call maven in batch mode

* Build shared module first

* Change compiled modules in release script

* Ask me to run mvn in windows

* Try building on self hosted runner again

* Run on all self-hosted

* Run on all self-hosted

* Run on all self-hosted

* Run on all self-hosted

* Translate build script to powershell

* Make maven quiet

* Escape -d options in maven

* Check exit code differently

* Ignore afile.txt

* Escape -D

* Allow "wet run" without release

* Add missing changelog.yaml

* Prepare for release

* Don't build shared in buildCore.cmd

* Wait for linux version after check of others

* Disable commit for test

* Build core from main folder

* Copy windows build artifacts to include folder

* Make wrapper open browser if main process can't

* Delete release shell script

* Fix build in system-test.yml

* Update changelog

* Update some libraries, ignore snakeyaml exploit in snyk

* Remove system test data

* Update to Spring Boot 3.0.2

* Differentiate versions of spring boot and devtools

* Set githubReleasesUrl directly
This commit is contained in:
theotherp 2023-01-20 15:02:00 +01:00 committed by GitHub
parent dc1cc469f7
commit 279b119f4c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1017 changed files with 102248 additions and 12050 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

59
.circleci/config.yml
View File

@ -1,59 +0,0 @@
# Java Maven CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-java/ for more details
#
version: 2
jobs:
build:
docker:
- image: circleci/openjdk:10-jdk-browsers
working_directory: ~/repo
environment:
- MAVEN_OPTS: -Xmx3200m
- PHANTOMJSBIN: /usr/local/bin/phantomjs
steps:
- checkout
- restore_cache:
keys:
- v1-dependencies-{{ checksum "pom.xml" }}
- v1-dependencies-
- run: mkdir -p ~/junit/
#Clean install all but without integration tests
- run: mvn -T 2 -pl "!org.nzbhydra:tests,!org.nzbhydra:linux-release,!org.nzbhydra:windows-release" clean install dependency:resolve-plugins dependency:go-offline
- run: find . -type f -regex ".*/target/surefire-reports/.*xml" -exec cp {} ~/junit/ \;
- run: find . -type f -regex ".*/target/screenshots/.*png" -exec cp {} ~/junit/ \;
# Integration tests produce weird transaction errors when run with maven :-/
# #Run integration tests
# - run: mvn -f tests/pom.xml test
# - run: find . -type f -regex ".*/target/surefire-reports/.*xml" -exec cp {} ~/junit/ \;
# - run: find . -type f -regex ".*/target/screenshots/.*png" -exec cp {} ~/junit/ \;
#Build releases
# - run: mkdir -p ~/releases/
# - run: mvn -T 2 -pl "org.nzbhydra:linux-release,org.nzbhydra:windows-release" clean install dependency:resolve-plugins dependency:go-offline
# - run: cp releases/linux-release/target/*.zip ~/releases/
# - run: cp releases/windows-release/target/*.zip ~/releases/
#Collect all surefire reports and put them in one folder for Circle to parse
- save_cache:
paths:
- ~/.m2
key: v1-dependencies-{{ checksum "pom.xml" }}
- store_test_results:
path: ~/junit
# - store_artifacts:
# path: ~/junit
# - store_artifacts:
# path: ~/releases
#workflows didn't work because I cached the m2 cache in the build step and then restored it in the test step. without a changed pom.xml old JARs would be used
#possible fix: another more specific save_cache for ~/.m2/repository/org/nzbhydra/

112
.github/workflows/buildNative.yml vendored Normal file
View File

@ -0,0 +1,112 @@
name: Native Build
on:
workflow_dispatch:
workflow_call:
jobs:
build:
strategy:
matrix:
os: [ ubuntu-20.04 ]
fail-fast: false
env:
HYDRA_NATIVE_BUILD: true
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v3
with:
# Check out last 15 commits
fetch-depth: 15
- name: Set up JDK 17
uses: actions/setup-java@v3
with:
java-version: '17'
distribution: 'adopt'
cache: 'maven'
- name: "Get changed files in core module"
id: changed-files-specific
uses: tj-actions/changed-files@v35
with:
since_last_remote_commit: true
files: |
core/**
- if: steps.changed-files-specific.outputs.any_changed == 'true'
run: |
echo "::notice::Will build new native image / docker container. Changed files in core since last push:"
echo "${{ steps.changed-files-specific.outputs.all_changed_files }}"
- if: steps.changed-files-specific.outputs.any_changed == 'false'
run: |
echo "::notice::Will skip build of native image / docker container. No changed files in core since last push."
- name: NativeImage
uses: graalvm/setup-graalvm@v1
if: steps.changed-files-specific.outputs.any_changed == 'true'
with:
java-version: '17'
version: 'latest'
components: 'native-image'
cache: 'maven'
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: "Install all with Maven"
run: mvn --batch-mode clean install -DskipTests -T 1C
- name: "Run unit tests"
run: mvn --batch-mode test -T 1C -pl !org.nzbhydra:tests,!org.nzbhydra.tests:system --fail-at-end
- name: "Create test Report"
uses: dorny/test-reporter@v1
if: always()
continue-on-error: true
with:
name: Unit test report
path: "**/surefire-reports/*.xml"
reporter: java-junit
- name: "Build native image"
if: steps.changed-files-specific.outputs.any_changed == 'true'
working-directory: ./core
run: |
mvn --batch-mode -Pnative clean native:compile -DskipTests
- name: "UPX linux artifact"
if: steps.changed-files-specific.outputs.any_changed == 'true'
uses: crazy-max/ghaction-upx@v2
with:
files: core/target/core
args: -q
- name: "Upload linux artifact"
if: steps.changed-files-specific.outputs.any_changed == 'true'
uses: actions/upload-artifact@v3
with:
name: coreLinux
path: core/target/core
- name: "Copy artifact to include folder"
if: steps.changed-files-specific.outputs.any_changed == 'true'
run: |
mv core/target/core ./docker/nativeTest/
chmod +x ./docker/nativeTest/core
- name: "Login to GitHub Container Registry"
if: steps.changed-files-specific.outputs.any_changed == 'true'
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: "Build core and push container"
if: steps.changed-files-specific.outputs.any_changed == 'true'
run: |
cp other/wrapper/nzbhydra2wrapperPy3.py ./docker/nativeTest/
cd ./docker/nativeTest/
docker build -t hydradocker .
docker tag hydradocker:latest ghcr.io/theotherp/hydradocker:latest
docker push ghcr.io/theotherp/hydradocker:latest

57
.github/workflows/release.yml vendored Normal file
View File

@ -0,0 +1,57 @@
name: Release
on:
workflow_dispatch:
inputs:
releaseVersion:
required: true
type: string
description: "Version to be released, like 1.2.3"
nextVersion:
required: true
type: string
description: "Version to be set afterwards, like 1.2.4-SNAPSHOT"
dryRun:
required: true
default: true
type: boolean
description: "Uncheck to actually execute the release"
selfHostedRunner:
required: true
default: false
type: boolean
description: "Has no effect, just as a reminder that the self-hosted windows runner must be running"
jobs:
build:
uses: ./.github/workflows/buildNative.yml
release:
needs: [build]
runs-on: ubuntu-latest
env:
githubReleasesUrl: https://api.github.com/repos/{{github.repository}}/releases
steps:
- uses: actions/checkout@v3
name: "Check out source"
- name: "Display structure of working directory"
run: ls .
- uses: actions/download-artifact@v3
name: "Download native artifacts"
with:
path: ~/artifacts
- name: "Display structure of artifacts folder"
run: ls -R ~/artifacts
- name: "Copy artifacts to include folders"
run: |
mv ~/artifacts/coreLinux/* ./releases/linux-release/include/
chmod +x ./releases/linux-release/include/nzbhydra2
mv ~/artifacts/coreWindows/* ./releases/windows-release/include/
- name: Set up JDK 17
uses: actions/setup-java@v3
with:
java-version: '17'
distribution: 'adopt'
cache: 'maven'
- name: "Run release script"
run: |
misc/build-and-release.sh ${{ github.event.inputs.releaseVersion }} ${{ github.event.inputs.nextVersion }} ${{ github.event.inputs.dryRun }}

130
.github/workflows/system-test.yml vendored Normal file
View File

@ -0,0 +1,130 @@
name: system-test
on:
push:
workflow_dispatch:
jobs:
waitForNative:
uses: ./.github/workflows/buildNative.yml
buildMockserver:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
name: "Check out source"
with:
# Check out last 15 commits
fetch-depth: 15
- name: "Get changed files in core module"
id: changed-files-specific
uses: tj-actions/changed-files@v35
with:
since_last_remote_commit: true
files: |
other/mockserver/**
- if: steps.changed-files-specific.outputs.any_changed == 'true'
run: |
echo "::notice::Will build new mock server container. Changed files since last push:"
echo "${{ steps.changed-files-specific.outputs.all_changed_files }}"
- if: steps.changed-files-specific.outputs.any_changed == 'false'
run: |
echo "::notice::Will skip build of new mock server container. No changed files since last push."
- name: Set up JDK 17
if: steps.changed-files-specific.outputs.any_changed == 'true'
uses: actions/setup-java@v3
with:
java-version: '17'
distribution: 'adopt'
cache: 'maven'
- name: "Login to GitHub Container Registry"
if: steps.changed-files-specific.outputs.any_changed == 'true'
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: "Build and push mockserver container"
if: steps.changed-files-specific.outputs.any_changed == 'true'
run: |
mvn --batch-mode install -DskipTests -T 1C
cd other/mockserver/
mvn --batch-mode spring-boot:build-image
docker tag mockserver:3.0.0 ghcr.io/theotherp/mockserver:3.0.0
docker push ghcr.io/theotherp/mockserver:3.0.0
runSystemTests:
needs: [ waitForNative, buildMockserver ]
runs-on: ubuntu-latest
strategy:
matrix:
test: [ { port: 5076, name: core }, { port: 5077, name: v1Migration } ]
env:
spring_profiles_active: build,systemtest
nzbhydra_port: ${{ matrix.test.port }}
nzbhydra_name: ${{ matrix.test.name }}
nzbhydra_host_external: http://${{ matrix.test.name }}:5076
nzbhydra.host.external: http://${{ matrix.test.name }}:5076
steps:
- run: echo Running test ${{ matrix.test.name }} with port ${{ matrix.test.port }}
- uses: actions/checkout@v3
name: "Check out source"
- name: Set up JDK 17
uses: actions/setup-java@v3
with:
java-version: '17'
distribution: 'adopt'
cache: 'maven'
- name: "Install"
run: mvn --batch-mode clean install -DskipTests -pl org.nzbhydra:nzbhydra2,org.nzbhydra:shared,org.nzbhydra:mapping,org.nzbhydra:assertions
- name: "Create docker network"
run: docker network create systemtest
- name: "Copy v1Migration docker data"
run: |
mkdir -p /tmp/hydra/v1MigrationDataFolder
cp -R tests/system/instanceData/v1Migration/* /tmp/hydra/v1MigrationDataFolder/
- name: "Run docker compose"
run: |
cd docker
docker-compose up -d
- name: "Wait for healthy containers"
run: |
docker ps
sleep 10
docker ps
sleep 10
docker ps
echo "Core container mounts:"
docker container inspect -f '{{ .Mounts}}' core
echo "v1Migration container mounts:"
docker container inspect -f '{{ .Mounts}}' v1Migration
- name: "Run tests"
run: mvn --batch-mode test -pl org.nzbhydra.tests:system -DtrimStackTrace=false
- name: "Upload data folder artifact"
uses: actions/upload-artifact@v3
if: always()
with:
name: data
path: /tmp/hydra
- name: "Create test Report"
uses: dorny/test-reporter@v1
if: always()
continue-on-error: true
with:
name: System test report ${{ matrix.test.name }}
path: "**/surefire-reports/*.xml"
reporter: java-junit

19
.github/workflows/test.yml vendored Normal file
View File

@ -0,0 +1,19 @@
name: Java CI
on: [ workflow_dispatch ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up JDK 17
uses: actions/setup-java@v3
with:
java-version: '17'
distribution: 'adopt'
cache: 'maven'
- name: Test with Maven
run: mvn --batch-mode --update-snapshots verify

2
.gitignore vendored
View File

@ -19,9 +19,11 @@ pom.xml.versionsBackup
javacore*
heapdump*
/ui
*token*
!/.idea/compiler.xml
!/.idea/vcs.xml
!/.idea/misc.xml
misc/rsyncToServers.sh
/nzbhydra.yml
/results

5
.idea/compiler.xml
View File

@ -22,7 +22,9 @@
<module name="mockserver" />
</profile>
</annotationProcessing>
<bytecodeTargetLevel target="8" />
<bytecodeTargetLevel target="17">
<module name="ui" target="1.5" />
</bytecodeTargetLevel>
</component>
<component name="EclipseCompilerSettings">
<option name="GENERATE_NO_WARNINGS" value="true" />
@ -31,7 +33,6 @@
<option name="ADDITIONAL_OPTIONS_STRING" value="-parameters" />
<option name="ADDITIONAL_OPTIONS_OVERRIDE">
<module name="core" options="-parameters" />
<module name="discordbot" options="" />
<module name="github-release-plugin" options="" />
<module name="linux-release" options="-parameters" />
<module name="mapping" options="-parameters" />

3
.idea/misc.xml
View File

@ -66,10 +66,11 @@
<option value="$PROJECT_DIR$/databaseperformance/pom.xml" />
<option value="$PROJECT_DIR$/pom.xml" />
<option value="$PROJECT_DIR$/shared/mapping/pom.xml" />
<option value="$PROJECT_DIR$/other/loadtest/pom.xml" />
</list>
</option>
</component>
<component name="ProjectRootManager" version="2" languageLevel="JDK_1_8" default="true" project-jdk-name="1.8" project-jdk-type="JavaSDK">
<component name="ProjectRootManager" version="2" languageLevel="JDK_17" project-jdk-name="17" project-jdk-type="JavaSDK">
<output url="file://$PROJECT_DIR$/out" />
</component>
</project>

19
.run/NzbHydraNativeEntrypoint.run.xml Normal file
View File

@ -0,0 +1,19 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="NzbHydraNativeEntrypoint" type="Application" factoryName="Application" nameIsGenerated="true">
<option name="ALTERNATIVE_JRE_PATH" value="17" />
<option name="ALTERNATIVE_JRE_PATH_ENABLED" value="true" />
<option name="MAIN_CLASS_NAME" value="org.nzbhydra.NzbHydraNativeEntrypoint" />
<module name="core" />
<option name="PROGRAM_PARAMETERS" value="org.nzbhydra.NzbHydraNative c:\Users\strat\IdeaProjects\nzbhydra2\core\target\spring-aot\main\sources c:\Users\strat\IdeaProjects\nzbhydra2\core\target\spring-aot\main\resources c:\Users\strat\IdeaProjects\nzbhydra2\core\target\spring-aot\main\classes\ org.nzbhydra nzbhydra2" />
<option name="WORKING_DIRECTORY" value="$PROJECT_DIR$/core" />
<extension name="coverage">
<pattern>
<option name="PATTERN" value="org.springframework.boot.*" />
<option name="ENABLED" value="true" />
</pattern>
</extension>
<method v="2">
<option name="Make" enabled="true" />
</method>
</configuration>
</component>

9
.snyk Normal file
View File

@ -0,0 +1,9 @@
# Snyk (https://snyk.io) policy file, patches or ignores known vulnerabilities.
version: v1.25.0
ignore:
SNYK-JAVA-ORGYAML-3152153:
- '*':
reason: no exploit
expires: 2030-04-01T00:00:00.000Z
created: 2023-01-20T10:45:42.937Z
patch: {}

12
buildCore.cmd Normal file
View File

@ -0,0 +1,12 @@
@echo off
setlocal
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat" x64
set path=c:\Programme\graalvm-ce-java17-22.2.0\bin\;%PATH%;c:\Programme\graalvm-ce-java17-22.2.0\bin\
set java_home=c:\Programme\graalvm-ce-java17-22.2.0\
set HYDRA_NATIVE_BUILD=true
call mvn -pl org.nzbhydra:core -Pnative clean native:compile -DskipTests
endlocal

26
changelog.md
View File

@ -48,13 +48,15 @@
### v4.7.0 BETA (2022-09-18)
**Feature** Use custom mappings to transform indexer result titles. Use this to clean up titles, add season or episode to it or whatever. See <a href="https://github.com/theotherp/nzbhydra2/issues/794">#794</a>
**Feature** Use custom customQueryAndTitleMappings to transform indexer result titles. Use this to clean up titles, add season or episode to it or whatever. See <a href="https://github.com/theotherp/nzbhydra2/issues/794">#794</a>
**Fix** Some of you have an instance running which is exposed to the internet, without any authentication method. I previously tried to recognize this by some heuristic which was a bit naive and caused a lot of false positives. NZBHydra will now periodically try to determine your public IP and actually check if the used port is open. This might still not always work (e.g. in when you're running it using a VPN in which case I guess you know what're doing. Ultimately it's up to you to get your shit together.
**Fix** Some of you have an instance running which is exposed to the internet, without any authentication method. I previously tried to recognize this by some heuristic which was a bit naive and caused a lot of false positives. NZBHydra
will now periodically try to determine your public IP and actually check if the used port is open. This might still not always work (e.g. in when you're running it using a VPN in which case I guess you know what're doing. Ultimately it's up
to you to get your shit together.
**Fix** Only warn about settings violating indexers' rules if the indexers are actually enabled.
**Fix** Fix saving config with custom mappings.
**Fix** Fix saving config with custom customQueryAndTitleMappings.
@ -74,7 +76,7 @@
**Feature** Automatically use NZB access and adding types required by certain indexers. See <a href="https://github.com/theotherp/nzbhydra2/issues/784">#784</a>.
**Feature** Add debug logging for category mapping.
**Feature** Add debug logging for category customQueryAndTitleMapping.
@ -100,13 +102,13 @@
**Fix** Add the current API hit to the number of reported API hits in response.
**Fix** Fix name of logging marker "Custom mapping" (was "Config mapping").
**Fix** Fix name of logging marker "Custom customQueryAndTitleMapping" (was "Config customQueryAndTitleMapping").
### v4.3.2 (2022-06-13)
**Fix** Fix use of groups in custom search request mapping. See <a href="https://github.com/theotherp/nzbhydra2/issues/700">#700</a>
**Fix** Fix use of groups in custom search request customQueryAndTitleMapping. See <a href="https://github.com/theotherp/nzbhydra2/issues/700">#700</a>
**Fix** Fix download of backup files. See <a href="https://github.com/theotherp/nzbhydra2/issues/772">#772</a>
@ -314,7 +316,8 @@
### v3.14.0 (2021-04-11)
**Feature** Custom mapping for queries and titles. This allows you to customize / change the values used by external tools or returned by metadata providers like TVDB. See <a href="https://github.com/theotherp/nzbhydra2/issues/700">#700</a>.
**Feature** Custom customQueryAndTitleMapping for queries and titles. This allows you to customize / change the values used by external tools or returned by metadata providers like TVDB.
See <a href="https://github.com/theotherp/nzbhydra2/issues/700">#700</a>.
@ -1366,7 +1369,8 @@
### v2.9.5 (2019-11-23)
**Feature** I realised the indexer score is too complex to show in a chart and replaced it with a table, that shows more information. It will now contain the average uniqueness score, the number of unique downloads and the number of searches which resulted in a download and where an indexer was involved.
**Feature** I realised the indexer score is too complex to show in a chart and replaced it with a table, that shows more information. It will now contain the average uniqueness score, the number of unique downloads and the number of
searches which resulted in a download and where an indexer was involved.
@ -1960,7 +1964,7 @@
### v2.1.7 (2018-12-30)
**Fix** Fix/improve category mapping introduced in 2.1.6. Use custom newznab categories if none from the predefined range are provided.
**Fix** Fix/improve category customQueryAndTitleMapping introduced in 2.1.6. Use custom newznab categories if none from the predefined range are provided.
@ -1968,7 +1972,7 @@
**Fix** When uploading a backup file the UI didn't update to inform the user about the progress after the file was uploaded.
**Fix** Improve category mapping for (torznab) indexers. Some use custom newznab category numbers (>9999) which could not be properly mapped to preconfigured categories.
**Fix** Improve category customQueryAndTitleMapping for (torznab) indexers. Some use custom newznab category numbers (>9999) which could not be properly mapped to preconfigured categories.
@ -2178,7 +2182,7 @@
**Fix** Restoring from web UI had no effect
**Fix** Category mapping would sometimes not work for incoming searches
**Fix** Category customQueryAndTitleMapping would sometimes not work for incoming searches

1
core/.gitignore vendored
View File

@ -23,3 +23,4 @@ sql notes.sql
*.dmp
*.trc
package-lock.json
afile.txt

12
core/buildCoreAotOnly.cmd Normal file
View File

@ -0,0 +1,12 @@
@echo off
setlocal
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat" x64
set path=c:\Programme\graalvm-ce-java17-22.2.0\bin\;%PATH%;c:\Programme\graalvm-ce-java17-22.2.0\bin\
set java_home=c:\Programme\graalvm-ce-java17-22.2.0\
set HYDRA_NATIVE_BUILD=true
mvn -Pnative clean package -DskipTests
endlocal

260
core/pom.xml
View File

@ -5,7 +5,7 @@
<parent>
<groupId>org.nzbhydra</groupId>
<artifactId>nzbhydra2</artifactId>
<version>4.7.7-SNAPSHOT</version>
<version>5.0.0-SNAPSHOT</version>
</parent>
<artifactId>core</artifactId>
@ -14,6 +14,7 @@
<!--Wrap in variable because otherwise it's not propagate to resource filtering below-->
<maven.build.timestamp2>${maven.build.timestamp}</maven.build.timestamp2>
<maven.build.timestamp.format>yyyy-MM-dd HH:mm</maven.build.timestamp.format>
<version.roaster>2.28.0.Final</version.roaster>
</properties>
<build>
@ -33,6 +34,16 @@
</excludes>
</resource>
</resources>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.graalvm.buildtools</groupId>
<artifactId>native-maven-plugin</artifactId>
<version>0.9.19</version>
<extensions>true</extensions>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
@ -46,6 +57,8 @@
</manifest>
<manifestEntries>
<Built-By>TheOtherP</Built-By>
<Version>${project.version}</Version>
<BuildTimestamp>${maven.build.timestamp}</BuildTimestamp>
</manifestEntries>
</archive>
<excludes>
@ -53,35 +66,30 @@
</excludes>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>${maven.compiler.plugin.version}</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>${spring.boot.version}</version>
<version>${spring.boot.maven.version}</version>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
<configuration>
<!--Add classifier to fat jar because otherwise depending modules won't find classes in the JAR-->
<classifier>exec</classifier>
<mainClass>org.nzbhydra.NzbHydra</mainClass>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M2</version>
<version>2.22.2</version>
</plugin>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.22.2</version>
</plugin>
</plugins>
</build>
@ -91,7 +99,7 @@
<dependency>
<groupId>org.nzbhydra</groupId>
<artifactId>mapping</artifactId>
<version>4.7.7-SNAPSHOT</version>
<version>5.0.0-SNAPSHOT</version>
</dependency>
<!-- spring (boot) -->
@ -163,13 +171,18 @@
<artifactId>spring-boot-starter-actuator</artifactId>
<version>${spring.boot.version}</version>
</dependency>
<dependency>
<groupId>org.glassfish.expressly</groupId>
<artifactId>expressly</artifactId>
<version>5.0.0</version>
</dependency>
<!-- database -->
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<version>1.4.200</version>
<version>2.1.214</version>
</dependency>
<dependency>
<groupId>com.github.marschall</groupId>
@ -197,7 +210,7 @@
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.13.0</version>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
@ -217,18 +230,19 @@
<!-- We use logback but due to a CVE in previous versions we force this dependency -->
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.17.2</version>
<version>2.19.0</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>${lombok.version}</version>
<optional>true</optional>
<scope>compile</scope>
<!-- <optional>true</optional>-->
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>20.0</version>
<version>31.1-jre</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
@ -240,11 +254,6 @@
<artifactId>commons-lang3</artifactId>
<version>${commons-lang3.version}</version>
</dependency>
<dependency>
<groupId>com.uwetrottmann.tmdb2</groupId>
<artifactId>tmdb-java</artifactId>
<version>1.6.0</version>
</dependency>
<dependency>
<groupId>com.github.briandilley.jsonrpc4j</groupId>
<artifactId>jsonrpc4j</artifactId>
@ -265,9 +274,9 @@
</exclusions>
</dependency>
<dependency>
<groupId>com.vladsch.flexmark</groupId>
<artifactId>flexmark</artifactId>
<version>0.34.12</version>
<groupId>org.commonmark</groupId>
<artifactId>commonmark</artifactId>
<version>0.20.0</version>
</dependency>
<dependency>
<groupId>com.squareup.okhttp3</groupId>
@ -277,7 +286,7 @@
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>logging-interceptor</artifactId>
<version>4.9.3</version>
<version>${okhttp.version}</version>
</dependency>
@ -289,7 +298,7 @@
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.11.3</version>
<version>1.15.3</version>
</dependency>
<dependency>
<groupId>net.jodah</groupId>
@ -301,54 +310,42 @@
<artifactId>URISchemeHandler</artifactId>
<version>${uri-scheme-handler.version}</version>
</dependency>
<dependency>
<groupId>com.sun.activation</groupId>
<artifactId>javax.activation</artifactId>
<version>1.2.0</version>
</dependency>
<dependency>
<groupId>com.github.ben-manes.caffeine</groupId>
<artifactId>caffeine</artifactId>
<version>2.6.2</version>
<version>3.1.2</version>
</dependency>
<dependency>
<groupId>net.jodah</groupId>
<groupId>dev.failsafe</groupId>
<artifactId>failsafe</artifactId>
<version>1.1.1</version>
<version>3.3.0</version>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.10.14</version>
<version>2.12.2</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.hibernate.validator</groupId>
<artifactId>hibernate-validator</artifactId>
<version>6.1.5.Final</version>
<version>8.0.0.Final</version>
</dependency>
<dependency>
<groupId>org.javers</groupId>
<artifactId>javers-core</artifactId>
<version>6.6.5</version>
<version>${javers-core.version}</version>
</dependency>
<dependency>
<!--https://stackoverflow.com/a/47412779-->
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<version>${spring.boot.version}</version>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springdoc</groupId>
<artifactId>springdoc-openapi-ui</artifactId>
<!-- When updating make sure the resource path in WebConfiguration is correct -->
<version>1.6.9</version>
</dependency>
<!-- todo spring only supports spring 2.x-->
<!-- <dependency>-->
<!-- <groupId>org.springdoc</groupId>-->
<!-- <artifactId>springdoc-openapi-ui</artifactId>-->
<!-- &lt;!&ndash; When updating make sure the resource path in WebConfiguration is correct &ndash;&gt;-->
<!-- <version>1.6.12</version>-->
<!-- </dependency>-->
<!--logging-->
<dependency>
@ -366,11 +363,12 @@
<artifactId>logback-access</artifactId>
<version>${logback.version}</version>
</dependency>
<dependency>
<groupId>net.rakugakibox.spring.boot</groupId>
<artifactId>logback-access-spring-boot-starter</artifactId>
<version>2.7.1</version>
</dependency>
<!-- todo spring Only supports Spring 2.x -->
<!-- <dependency>-->
<!-- <groupId>net.rakugakibox.spring.boot</groupId>-->
<!-- <artifactId>logback-access-spring-boot-starter</artifactId>-->
<!-- <version>2.7.1</version>-->
<!-- </dependency>-->
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
@ -385,17 +383,17 @@
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.36</version>
<version>2.0.5</version>
</dependency>
<dependency>
<groupId>org.nzbhydra</groupId>
<artifactId>sockslib</artifactId>
<version>1.0.0</version>
<version>3.0.0</version>
</dependency>
<dependency>
<groupId>net.sourceforge.htmlunit</groupId>
<artifactId>htmlunit</artifactId>
<version>2.64.0</version>
<version>2.67.0</version>
</dependency>
@ -420,13 +418,13 @@
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>4.5.1</version>
<version>4.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-library</artifactId>
<version>1.3</version>
<version>2.2</version>
<scope>test</scope>
</dependency>
<dependency>
@ -442,13 +440,139 @@
</exclusions>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
<groupId>org.jboss.forge.roaster</groupId>
<artifactId>roaster-api</artifactId>
<version>${version.roaster}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jboss.forge.roaster</groupId>
<artifactId>roaster-jdt</artifactId>
<version>${version.roaster}</version>
<scope>test</scope>
</dependency>
</dependencies>
<profiles>
<profile>
<id>dev</id>
<dependencies>
<dependency>
<!--https://stackoverflow.com/a/47412779-->
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<version>${spring.boot.devtools.version}</version>
<optional>true</optional>
</dependency>
</dependencies>
</profile>
<profile>
<id>native</id>
<properties>
<start-class>org.nzbhydra.NzbHydra</start-class>
</properties>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<mainClass>org.nzbhydra.NzbHydra</mainClass>
<image>
<builder>paketobuildpacks/builder:tiny</builder>
<env>
<BP_NATIVE_IMAGE>true</BP_NATIVE_IMAGE>
</env>
</image>
</configuration>
<executions>
<execution>
<id>process-aot</id>
<goals>
<goal>process-aot</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.graalvm.buildtools</groupId>
<artifactId>native-maven-plugin</artifactId>
<configuration>
<buildArgs>
<buildArg>-H:+ReportExceptionStackTraces</buildArg>
<buildArg>-H:-DeadlockWatchdogExitOnTimeout</buildArg>
<buildArg>-H:DeadlockWatchdogInterval=0</buildArg>
<buildArg>--initialize-at-build-time=org.apache.commons.logging.LogFactoryService</buildArg>
</buildArgs>
<classesDirectory>${project.build.outputDirectory}</classesDirectory>
<metadataRepository>
<enabled>true</enabled>
</metadataRepository>
<requiredVersion>22.3</requiredVersion>
</configuration>
<executions>
<execution>
<id>add-reachability-metadata</id>
<goals>
<goal>add-reachability-metadata</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</pluginManagement>
</build>
</profile>
<profile>
<id>nativeTest</id>
<!-- <dependencies>-->
<!-- <dependency>-->
<!-- <groupId>org.junit.platform</groupId>-->
<!-- <artifactId>junit-platform-launcher</artifactId>-->
<!-- <version>1.9.1</version>-->
<!-- <scope>test</scope>-->
<!-- </dependency>-->
<!-- </dependencies>-->
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<executions>
<execution>
<id>process-test-aot</id>
<goals>
<goal>process-test-aot</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.graalvm.buildtools</groupId>
<artifactId>native-maven-plugin</artifactId>
<configuration>
<classesDirectory>${project.build.outputDirectory}</classesDirectory>
<metadataRepository>
<enabled>true</enabled>
</metadataRepository>
<requiredVersion>22.3</requiredVersion>
</configuration>
<executions>
<execution>
<id>native-test</id>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>

13
core/runTracingAgent.cmd Normal file
View File

@ -0,0 +1,13 @@
@echo off
setlocal
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat" x64
set path=c:\Programme\graalvm-ce-java17-22.2.0\bin\;%PATH%;c:\Programme\graalvm-ce-java17-22.2.0\bin\
set java_home=c:\Programme\graalvm-ce-java17-22.2.0\
set HYDRA_NATIVE_BUILD=true
cd target
java -DspringAot=true -agentlib:native-image-agent=config-output-dir=hints -jar core-5.0.0-SNAPSHOT-exec.jar directstart
endlocal

18
core/src/main/java/org/nzbhydra/DevEndpoint.java
View File

@ -16,6 +16,8 @@
package org.nzbhydra;
import jakarta.persistence.EntityManager;
import jakarta.persistence.PersistenceContext;
import org.nzbhydra.config.ConfigProvider;
import org.nzbhydra.config.indexer.IndexerConfig;
import org.nzbhydra.externaltools.AddRequest;
@ -25,14 +27,14 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.ApplicationEventPublisher;
import org.springframework.http.MediaType;
import org.springframework.security.access.AccessDeniedException;
import org.springframework.security.access.annotation.Secured;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;
import javax.persistence.EntityManager;
import javax.persistence.PersistenceContext;
import java.math.BigInteger;
import java.util.List;
@ -58,6 +60,18 @@ public class DevEndpoint {
return resultList.get(0);
}
@Secured({"ROLE_ADMIN"})
@RequestMapping(value = "/dev/throwException", method = RequestMethod.GET)
public BigInteger throwException() throws Exception {
throw new RuntimeException("test");
}
@Secured({"ROLE_ADMIN"})
@RequestMapping(value = "/dev/throwAccessDeniedException", method = RequestMethod.POST, produces = MediaType.APPLICATION_JSON_VALUE, consumes = MediaType.APPLICATION_JSON_VALUE)
public BigInteger throwAccessDeniedException() throws Exception {
throw new AccessDeniedException("test");
}
@Secured({"ROLE_ADMIN"})
@Transactional
@RequestMapping(value = "/dev/deleteDanglingIndexersearches", method = RequestMethod.GET)

2
core/src/main/java/org/nzbhydra/ExceptionInfo.java
View File

@ -1,10 +1,12 @@
package org.nzbhydra;
import lombok.Data;
import org.nzbhydra.springnative.ReflectionMarker;
import java.time.Instant;
@Data
@ReflectionMarker
public class ExceptionInfo {
private long timestamp;
private int status;

7
core/src/main/java/org/nzbhydra/InstanceCounter.java
View File

@ -16,6 +16,8 @@
package org.nzbhydra;
import jakarta.annotation.PostConstruct;
import org.nzbhydra.config.BaseConfigHandler;
import org.nzbhydra.config.ConfigProvider;
import org.nzbhydra.webaccess.HydraOkHttp3ClientHttpRequestFactory;
import org.slf4j.Logger;
@ -26,7 +28,6 @@ import org.springframework.http.client.ClientHttpRequest;
import org.springframework.http.client.ClientHttpResponse;
import org.springframework.stereotype.Component;
import javax.annotation.PostConstruct;
import java.io.IOException;
import java.net.URI;
@ -42,6 +43,8 @@ public class InstanceCounter {
@Autowired
private ConfigProvider configProvider;
@Autowired
private BaseConfigHandler baseConfigHandler;
@PostConstruct
public void downloadInstanceCounter() {
@ -51,7 +54,7 @@ public class InstanceCounter {
if (response.getStatusCode().is2xxSuccessful()) {
logger.info("Instance counted");
configProvider.getBaseConfig().getMain().setInstanceCounterDownloaded(true);
configProvider.getBaseConfig().save(false);
baseConfigHandler.save(false);
} else {
logger.error("Unable to count instance. Response: " + response.getStatusText());
}

13
core/src/main/java/org/nzbhydra/Markdown.java
View File

@ -1,17 +1,16 @@
package org.nzbhydra;
import com.vladsch.flexmark.ast.Node;
import com.vladsch.flexmark.html.HtmlRenderer;
import com.vladsch.flexmark.parser.Parser;
import com.vladsch.flexmark.util.options.MutableDataSet;
import org.commonmark.node.Node;
import org.commonmark.parser.Parser;
import org.commonmark.renderer.html.HtmlRenderer;
public class Markdown {
public static String renderMarkdownAsHtml(String markdown) {
MutableDataSet options = new MutableDataSet();
Parser parser = Parser.builder(options).build();
HtmlRenderer renderer = HtmlRenderer.builder(options).build();
Parser parser = Parser.builder().build();
Node document = parser.parse(markdown);
HtmlRenderer renderer = HtmlRenderer.builder().build();
return renderer.render(document);
}

69
core/src/main/java/org/nzbhydra/NativeHints.java Normal file
View File

@ -0,0 +1,69 @@
/*
* (C) Copyright 2023 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra;
import org.commonmark.renderer.html.HtmlRenderer;
import org.nzbhydra.config.migration.ConfigMigrationStep;
import org.nzbhydra.springnative.ReflectionMarker;
import org.reflections.Reflections;
import org.reflections.scanners.Scanners;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.aot.hint.ExecutableMode;
import org.springframework.aot.hint.MemberCategory;
import org.springframework.aot.hint.RuntimeHints;
import org.springframework.aot.hint.RuntimeHintsRegistrar;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.Set;
public class NativeHints implements RuntimeHintsRegistrar {
private static final Logger logger = LoggerFactory.getLogger(NativeHints.class);
@Override
public void registerHints(RuntimeHints hints, ClassLoader classLoader) {
logger.info("Registering native hints");
hints.resources().registerResourceBundle("joptsimple.ExceptionMessages");
final Set<Class<?>> classes = getClassesToRegister();
classes.add(HashSet.class);
classes.add(ArrayList.class);
classes.add(HtmlRenderer.class);
for (Class<?> clazz : classes) {
hints.reflection().registerType(clazz, MemberCategory.INVOKE_PUBLIC_CONSTRUCTORS);
for (Method method : clazz.getDeclaredMethods()) {
logger.info("Registering " + method + " for reflection");
hints.reflection().registerMethod(method, ExecutableMode.INVOKE);
}
}
}
private static Set<Class<?>> getClassesToRegister() {
final Reflections reflections = new Reflections("org.nzbhydra", Scanners.TypesAnnotated, Scanners.SubTypes);
final Set<Class<?>> classes = reflections.getTypesAnnotatedWith(ReflectionMarker.class);
classes.addAll(reflections.getSubTypesOf(ConfigMigrationStep.class));
return classes;
}
}

166
core/src/main/java/org/nzbhydra/NzbHydra.java
View File

@ -2,19 +2,21 @@ package org.nzbhydra;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.google.common.base.Strings;
import jakarta.annotation.PostConstruct;
import jakarta.annotation.PreDestroy;
import joptsimple.OptionException;
import joptsimple.OptionParser;
import joptsimple.OptionSet;
import org.apache.commons.lang3.tuple.Pair;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.config.BaseConfigHandler;
import org.nzbhydra.config.ConfigProvider;
import org.nzbhydra.config.ConfigReaderWriter;
import org.nzbhydra.config.migration.ConfigMigration;
import org.nzbhydra.database.DatabaseRecreation;
import org.nzbhydra.debuginfos.DebugInfosProvider;
import org.nzbhydra.genericstorage.GenericStorage;
import org.nzbhydra.logging.LoggingMarkers;
import org.nzbhydra.misc.BrowserOpener;
import org.nzbhydra.update.UpdateManager;
import org.nzbhydra.web.UrlCalculator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@ -33,28 +35,23 @@ import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.ImportRuntimeHints;
import org.springframework.context.event.EventListener;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.transaction.annotation.EnableTransactionManagement;
import org.springframework.web.bind.annotation.RestController;
import org.yaml.snakeyaml.error.YAMLException;
import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import javax.swing.*;
import java.awt.GraphicsEnvironment;
import java.awt.HeadlessException;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.PrintStream;
import java.net.URI;
import java.nio.file.Files;
import java.time.LocalDateTime;
import java.util.Arrays;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
@ImportRuntimeHints(NativeHints.class)
@Configuration(proxyBeanMethods = false)
@EnableAutoConfiguration(exclude = {
AopAutoConfiguration.class, org.springframework.boot.autoconfigure.cache.CacheAutoConfiguration.class})
@ -73,7 +70,7 @@ public class NzbHydra {
private static String dataFolder = null;
private static boolean wasRestarted = false;
private static boolean anySettingsOverwritten = false;
private static ConfigReaderWriter configReaderWriter = new ConfigReaderWriter();
private static final ConfigReaderWriter CONFIG_READER_WRITER = new ConfigReaderWriter();
@Autowired
private ConfigProvider configProvider;
@ -85,12 +82,29 @@ public class NzbHydra {
private GenericStorage genericStorage;
@Autowired
private ApplicationEventPublisher applicationEventPublisher;
@Autowired
private BaseConfigHandler baseConfigHandler;
@Autowired
private DebugInfosProvider debugInfosProvider;
public static void main(String[] args) throws Exception {
LoggerFactory.getILoggerFactory();
if (isNativeBuild()) {
logger.warn("Running for native build");
String dataFolder = "./data";
NzbHydra.setDataFolder(dataFolder);
System.setProperty("nzbhydra.dataFolder", dataFolder);
System.setProperty("spring.datasource.url", "jdbc:h2:mem:testdb;NON_KEYWORDS=YEAR,DATA,KEY");
checkJavaVersion();
setApplicationPropertiesFromConfig();
SpringApplication hydraApplication = new SpringApplication(NzbHydra.class);
applicationContext = hydraApplication.run(args);
logger.info("Native application returned");
return;
}
OptionParser parser = new OptionParser();
parser.accepts("datafolder", "Define path to main data folder. Must start with ./ for relative paths").withRequiredArg().defaultsTo("./data");
@ -108,46 +122,22 @@ public class NzbHydra {
logger.error("Invalid startup options detected: {}", e.getMessage());
System.exit(1);
}
if (System.getProperty("fromWrapper") == null && Arrays.stream(args).noneMatch(x -> x.equals("directstart"))) {
logger.info("NZBHydra 2 must be started using the wrapper for restart and updates to work. If for some reason you need to start it from the JAR directly provide the command line argument \"directstart\"");
} else if (options.has("help")) {
setDataFolder(options);
if (options.has("help")) {
parser.printHelpOn(System.out);
} else if (options.has("version")) {
String version = new UpdateManager().getAllVersionChangesUpToCurrentVersion().get(0).getVersion();
logger.info("NZBHydra 2 version: " + version);
System.out.println(DebugInfosProvider.getVersionAndBuildTimestamp().getLeft());
} else if (System.getProperty("fromWrapper") == null && Arrays.stream(args).noneMatch(x -> x.equals("directstart"))) {
logger.info("NZBHydra 2 must be started using the wrapper for restart and updates to work. If for some reason you need to start it from the JAR directly provide the command line argument \"directstart\"");
} else {
startup(args, options);
}
}
private static void checkJavaVersion() {
final String javaVersionString;
int javaMajor = 0;
try {
javaVersionString = System.getProperty("java.version");
final Matcher matcher = Pattern.compile("(?<major>\\d+)(\\.(?<minor>\\d+)\\.(?<patch>\\d)+[\\-_\\w]*)?.*").matcher(javaVersionString);
if (!matcher.find()) {
logger.error("Unable to determine JAVA version from {}", javaVersionString);
return;
}
javaMajor = Integer.parseInt(matcher.group("major"));
int javaMinor = Integer.parseInt(matcher.group("minor"));
int javaVersion = 0;
if ((javaMajor == 1 && javaMinor == 8)) {
return;
}
} catch (Exception e) {
logger.error("Unable to determine java version", e);
return;
}
if (javaMajor > 17) {
throw new RuntimeException("Deteted Java version " + javaVersionString + ". Please use Java 8, 11, 15 or 17. Java 18 and above are not supported");
}
}
protected static void startup(String[] args, OptionSet options) throws Exception {
private static void setDataFolder(OptionSet options) throws IOException {
if (options.has("datafolder")) {
dataFolder = (String) options.valueOf("datafolder");
} else {
@ -155,6 +145,10 @@ public class NzbHydra {
}
File dataFolderFile = new File(dataFolder);
dataFolder = dataFolderFile.getCanonicalPath();
}
protected static void startup(String[] args, OptionSet options) throws Exception {
File dataFolderFile = new File(dataFolder);
//Check if we can write in the data folder. If not we can just quit now
if (!dataFolderFile.exists() && !dataFolderFile.mkdirs()) {
logger.error("Unable to read or write data folder {}", dataFolder);
@ -185,26 +179,22 @@ public class NzbHydra {
SpringApplication hydraApplication = new SpringApplication(NzbHydra.class);
NzbHydra.originalArgs = args;
wasRestarted = Arrays.asList(args).contains("restarted");
if (NzbHydra.isOsWindows() && !options.has("quiet") && !options.has("nobrowser")) {
hydraApplication.setHeadless(false);
} else {
hydraApplication.setHeadless(true);
}
DatabaseRecreation.runDatabaseScript();
applicationContext = hydraApplication.run(args);
} catch (Exception e) {
//Is thrown by SpringApplicationAotProcessor
if (!(e instanceof SpringApplication.AbandonedRunException)) {
handleException(e);
}
}
}
/**
* Sets all properties referenced in application.properties so that they can be resolved
*/
private static void setApplicationPropertiesFromConfig() throws IOException {
BaseConfig baseConfig = configReaderWriter.loadSavedConfig();
BaseConfig baseConfig = CONFIG_READER_WRITER.loadSavedConfig();
setApplicationProperty("main.host", "MAIN_HOST", baseConfig.getMain().getHost());
setApplicationProperty("main.port", "MAIN_PORT", String.valueOf(baseConfig.getMain().getPort()));
setApplicationProperty("main.urlBase", "MAIN_URL_BASE", baseConfig.getMain().getUrlBase().orElse("/"));
@ -230,9 +220,9 @@ public class NzbHydra {
File systemErrLogFile = new File(NzbHydra.getDataFolder(), "logs/system.err.log");
File systemOutLogFile = new File(NzbHydra.getDataFolder(), "logs/system.out.log");
logger.info("Enabling SSL debugging. Will write to {}", systemErrLogFile);
System.setErr(new PrintStream(new FileOutputStream(systemErrLogFile)));
System.setErr(new PrintStream(Files.newOutputStream(systemErrLogFile.toPath())));
logger.info("Redirecting console output to system.out.log. You will not see any more log output in the console until you disable the HTTPS marker and restart NZBHydra");
System.setOut(new PrintStream(new FileOutputStream(systemOutLogFile)));
System.setOut(new PrintStream(Files.newOutputStream(systemOutLogFile.toPath())));
}
}
}
@ -245,11 +235,14 @@ public class NzbHydra {
}
private static void initializeAndValidateAndMigrateYamlFile(File yamlFile) throws IOException {
configReaderWriter.initializeIfNeeded(yamlFile);
configReaderWriter.validateExistingConfig();
Map<String, Object> map = configReaderWriter.loadSavedConfigAsMap();
if (NzbHydra.isNativeBuild()) {
return;
}
CONFIG_READER_WRITER.initializeIfNeeded(yamlFile);
CONFIG_READER_WRITER.validateExistingConfig();
Map<String, Object> map = CONFIG_READER_WRITER.loadSavedConfigAsMap();
Map<String, Object> migrated = new ConfigMigration().migrate(map);
configReaderWriter.save(migrated, yamlFile);
CONFIG_READER_WRITER.save(migrated, yamlFile);
}
private static void handleException(Exception e) throws Exception {
@ -272,39 +265,19 @@ public class NzbHydra {
msg = "An unexpected error occurred during startup:\n" + e;
logger.error("An unexpected error occurred during startup", e);
}
try {
if (!GraphicsEnvironment.isHeadless() && isOsWindows()) {
final String htmlMessage = "<html>" + msg.replace("\n", "<br>") + "</html>";
JOptionPane.showMessageDialog(null, htmlMessage, "NZBHydra 2 error", JOptionPane.ERROR_MESSAGE);
}
} catch (HeadlessException e1) {
logger.warn("Unable to show exception in message dialog: {}", e1.getMessage());
}
logger.error("FATAL: " + msg, e);
//Rethrow so that spring exception handlers can handle this
throw e;
}
@PostConstruct
private void addTrayIconIfApplicable() {
boolean isOsWindows = isOsWindows();
if (isOsWindows) {
logger.info("Adding windows system tray icon");
try {
new WindowsTrayIcon();
} catch (Throwable e) {
logger.error("Can't add a windows tray icon because running headless");
}
}
}
public static boolean isOsWindows() {
String osName = System.getProperty("os.name");
return osName.toLowerCase().contains("windows");
}
@PostConstruct
private void warnIfSettingsOverwritten() {
public void warnIfSettingsOverwritten() {
if (anySettingsOverwritten) {
logger.warn("Overwritten settings will be displayed with their original value in the config section of the GUI");
}
@ -338,21 +311,26 @@ public class NzbHydra {
//I don't know why I have to do this but otherwise genericStorage is always empty
configProvider.getBaseConfig().setGenericStorage(new ConfigReaderWriter().loadSavedConfig().getGenericStorage());
if (!genericStorage.get("FirstStart", LocalDateTime.class).isPresent()) {
final Pair<String, String> versionAndBuildTimestamp = DebugInfosProvider.getVersionAndBuildTimestamp();
logger.info("Version: {}", versionAndBuildTimestamp.getLeft());
logger.info("Build timestamp: {}", versionAndBuildTimestamp.getRight());
if (genericStorage.get("FirstStart", LocalDateTime.class).isEmpty()) {
logger.info("First start of NZBHydra detected");
genericStorage.save("FirstStart", LocalDateTime.now());
configProvider.getBaseConfig().save(false);
baseConfigHandler.save(false);
}
if (DebugInfosProvider.isRunInDocker()) {
logger.info("You seem to be running NZBHydra 2 in docker. You can access Hydra using your local address and the IP you provided");
} else if (configProvider.getBaseConfig().getMain().isStartupBrowser() && !"true".equals(System.getProperty(BROWSER_DISABLED))) {
} else {
if (configProvider.getBaseConfig().getMain().isStartupBrowser() && !"true".equals(System.getProperty(BROWSER_DISABLED))) {
if (wasRestarted) {
logger.info("Not opening browser after restart");
return;
}
browserOpener.openBrowser();
} else {
}
URI uri = urlCalculator.getLocalBaseUriBuilder().build().toUri();
logger.info("You can access NZBHydra 2 in your browser via {}", uri);
}
@ -363,16 +341,6 @@ public class NzbHydra {
@PreDestroy
public void destroy() {
boolean isOsWindows = isOsWindows();
if (isOsWindows) {
logger.debug("Initiating removal of windows tray icon (if it exists)");
try {
WindowsTrayIcon.remove();
} catch (Throwable e) {
//An exception might be thrown while shutting down, ignore this
}
}
applicationEventPublisher.publishEvent(new ShutdownEvent());
logger.info("Shutting down and using up to {}ms to compact database", configProvider.getBaseConfig().getMain().getDatabaseCompactTime());
}
@ -382,5 +350,11 @@ public class NzbHydra {
return new CaffeineCacheManager("infos", "titles", "updates", "dev");
}
static void setDataFolder(String dataFolder) {
NzbHydra.dataFolder = dataFolder;
}
public static boolean isNativeBuild() {
return System.getenv("HYDRA_NATIVE_BUILD") != null;
}
}

48
core/src/main/java/org/nzbhydra/NzbHydraNativeEntrypoint.java Normal file
View File

@ -0,0 +1,48 @@
/*
* (C) Copyright 2022 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra;
import jakarta.annotation.PostConstruct;
import org.springframework.boot.SpringApplicationAotProcessor;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.aot.AbstractAotProcessor;
import org.springframework.javapoet.ClassName;
import org.springframework.util.Assert;
import org.springframework.util.StringUtils;
import java.nio.file.Paths;
import java.util.Arrays;
@Configuration
public class NzbHydraNativeEntrypoint {
public static void main(String[] args) throws Exception {
int requiredArgs = 6;
Assert.isTrue(args.length >= requiredArgs, () -> "Usage: " + SpringApplicationAotProcessor.class.getName()
+ " <applicationName> <sourceOutput> <resourceOutput> <classOutput> <groupId> <artifactId> <originalArgs...>");
Class<?> application = Class.forName(args[0]);
AbstractAotProcessor.Settings settings = AbstractAotProcessor.Settings.builder().sourceOutput(Paths.get(args[1])).resourceOutput(Paths.get(args[2]))
.classOutput(Paths.get(args[3])).groupId((StringUtils.hasText(args[4])) ? args[4] : "unspecified")
.artifactId(args[5]).build();
String[] applicationArgs = (args.length > requiredArgs) ? Arrays.copyOfRange(args, requiredArgs, args.length)
: new String[0];
final SpringApplicationAotProcessor processor = new SpringApplicationAotProcessor(application, settings, applicationArgs);
final ClassName process = processor.process();
System.exit(0);
}
}

123
core/src/main/java/org/nzbhydra/WindowsTrayIcon.java
View File

@ -1,123 +0,0 @@
package org.nzbhydra;
import org.nzbhydra.misc.BrowserOpener;
import org.nzbhydra.systemcontrol.SystemControl;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.BeansException;
import org.springframework.context.ConfigurableApplicationContext;
import javax.swing.*;
import java.awt.AWTException;
import java.awt.MenuItem;
import java.awt.PopupMenu;
import java.awt.SystemTray;
import java.awt.TrayIcon;
import java.awt.event.MouseEvent;
import java.awt.event.MouseListener;
//Mostly taken from https://stackoverflow.com/a/44452260
public class WindowsTrayIcon extends TrayIcon {
private static final Logger logger = LoggerFactory.getLogger(WindowsTrayIcon.class);
private static final String IMAGE_PATH = "/nzbhydra.png";
private static final String TOOLTIP = "NZBHydra 2";
private PopupMenu popup;
private static SystemTray tray;
private static WindowsTrayIcon instance;
public WindowsTrayIcon() {
super(new ImageIcon(WindowsTrayIcon.class.getResource(IMAGE_PATH), TOOLTIP).getImage(), TOOLTIP);
try {
popup = new PopupMenu();
tray = SystemTray.getSystemTray();
instance = this;
setup();
} catch (AWTException e) {
logger.error("Unable to create tray icon", e);
}
}
private void setup() throws AWTException {
MenuItem openBrowserItem = new MenuItem("Open web UI");
popup.add(openBrowserItem);
openBrowserItem.addActionListener(e -> {
openBrowser();
});
MenuItem restartItem = new MenuItem("Restart");
popup.add(restartItem);
restartItem.addActionListener(e -> {
((ConfigurableApplicationContext) NzbHydra.getApplicationContext()).close();
remove();
System.exit(SystemControl.RESTART_RETURN_CODE);
});
MenuItem shutdownItem = new MenuItem("Shutdown");
popup.add(shutdownItem);
shutdownItem.addActionListener(e -> {
try {
((ConfigurableApplicationContext) NzbHydra.getApplicationContext()).close();
} catch (Exception e1) {
logger.error("Error while closing application context, will shut down hard");
} finally {
remove();
}
System.exit(SystemControl.SHUTDOWN_RETURN_CODE);
});
setPopupMenu(popup);
tray.add(this);
instance.addMouseListener(new MouseListener() {
@Override
public void mouseClicked(MouseEvent e) {
if (e.getClickCount() == 2) {
openBrowser();
}
}
@Override
public void mousePressed(MouseEvent e) {
}
@Override
public void mouseReleased(MouseEvent e) {
}
@Override
public void mouseEntered(MouseEvent e) {
}
@Override
public void mouseExited(MouseEvent e) {
}
});
}
private void openBrowser() {
try {
NzbHydra.getApplicationContext().getAutowireCapableBeanFactory().createBean(BrowserOpener.class).openBrowser();
} catch (NullPointerException | IllegalStateException | BeansException e1) {
logger.error("Unable to open browser. Process may not have started completely");
}
}
public static void remove() {
try {
if (tray != null && instance != null) {
logger.info("Removing tray icon");
tray.remove(instance);
tray = null;
}
} catch (Throwable e) {
logger.error("Unable to remove tray icon", e);
}
}
}

9
core/src/main/java/org/nzbhydra/api/CapsGenerator.java
View File

@ -18,10 +18,12 @@ package org.nzbhydra.api;
import com.google.common.collect.Sets;
import org.nzbhydra.config.ConfigProvider;
import org.nzbhydra.config.SearchSource;
import org.nzbhydra.config.SearchSourceRestriction;
import org.nzbhydra.config.category.Category;
import org.nzbhydra.config.indexer.IndexerConfig;
import org.nzbhydra.config.indexer.SearchModuleType;
import org.nzbhydra.config.mediainfo.MediaIdType;
import org.nzbhydra.mapping.newznab.ActionAttribute;
import org.nzbhydra.mapping.newznab.NewznabResponse;
import org.nzbhydra.mapping.newznab.OutputType;
@ -47,8 +49,6 @@ import org.nzbhydra.mapping.newznab.xml.caps.CapsXmlSearch;
import org.nzbhydra.mapping.newznab.xml.caps.CapsXmlSearching;
import org.nzbhydra.mapping.newznab.xml.caps.CapsXmlServer;
import org.nzbhydra.mediainfo.InfoProvider;
import org.nzbhydra.mediainfo.MediaIdType;
import org.nzbhydra.searching.searchrequests.SearchRequest;
import org.nzbhydra.update.UpdateManager;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpHeaders;
@ -66,7 +66,6 @@ import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;
@Component
public class CapsGenerator {
@ -174,7 +173,7 @@ public class CapsGenerator {
if (x.getState() != IndexerConfig.State.ENABLED && x.getState() != IndexerConfig.State.DISABLED_SYSTEM_TEMPORARY) {
return false;
}
if (!x.getEnabledForSearchSource().meets(SearchRequest.SearchSource.API)) {
if (!SearchSource.API.meets(x.getEnabledForSearchSource())) {
//Indexer will not be picked for API searches
return false;
}
@ -219,7 +218,7 @@ public class CapsGenerator {
List<Integer> subCategories = category.getNewznabCategories().stream().flatMap(Collection::stream).filter(x -> x % 1000 != 0)
//Lower numbers first so that predefined category numbers take precedence over custom ones
.sorted(Comparator.naturalOrder())
.collect(Collectors.toList());
.toList();
//Use lowest category first
for (Integer subCategory : subCategories) {
if (alreadyAdded.contains(category.getName())) {

5
core/src/main/java/org/nzbhydra/api/CategoryConverter.java
View File

@ -1,13 +1,12 @@
package org.nzbhydra.api;
import jakarta.persistence.AttributeConverter;
import jakarta.persistence.Converter;
import org.nzbhydra.config.category.Category;
import org.nzbhydra.searching.CategoryProvider;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import javax.persistence.AttributeConverter;
import javax.persistence.Converter;
@Converter
@Component

16
core/src/main/java/org/nzbhydra/api/ExternalApi.java
View File

@ -8,7 +8,11 @@ import lombok.AllArgsConstructor;
import lombok.Data;
import org.apache.catalina.connector.ClientAbortException;
import org.nzbhydra.config.ConfigProvider;
import org.nzbhydra.config.SearchSource;
import org.nzbhydra.config.category.CategoriesConfig;
import org.nzbhydra.config.downloading.DownloadType;
import org.nzbhydra.config.mediainfo.MediaIdType;
import org.nzbhydra.config.searching.SearchType;
import org.nzbhydra.downloading.DownloadResult;
import org.nzbhydra.downloading.FileHandler;
import org.nzbhydra.downloading.InvalidSearchResultIdException;
@ -19,16 +23,13 @@ import org.nzbhydra.mapping.newznab.NewznabResponse;
import org.nzbhydra.mapping.newznab.OutputType;
import org.nzbhydra.mapping.newznab.xml.NewznabXmlError;
import org.nzbhydra.mediainfo.Imdb;
import org.nzbhydra.mediainfo.MediaIdType;
import org.nzbhydra.searching.CategoryProvider;
import org.nzbhydra.searching.CustomQueryAndTitleMapping;
import org.nzbhydra.searching.CustomQueryAndTitleMappingHandler;
import org.nzbhydra.searching.SearchResult;
import org.nzbhydra.searching.Searcher;
import org.nzbhydra.searching.dtoseventsenums.DownloadType;
import org.nzbhydra.searching.dtoseventsenums.SearchType;
import org.nzbhydra.searching.searchrequests.SearchRequest;
import org.nzbhydra.searching.searchrequests.SearchRequest.SearchSource;
import org.nzbhydra.searching.searchrequests.SearchRequestFactory;
import org.nzbhydra.springnative.ReflectionMarker;
import org.nzbhydra.web.SessionStorage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@ -90,7 +91,7 @@ public class ExternalApi {
@Autowired
private MockSearch mockSearch;
@Autowired
private CustomQueryAndTitleMapping customQueryAndTitleMapping;
private CustomQueryAndTitleMappingHandler customQueryAndTitleMappingHandler;
protected Clock clock = Clock.systemUTC();
private final Random random = new Random();
@ -336,13 +337,14 @@ public class ExternalApi {
searchRequest.getInternalData().setIncludePasswords(true);
}
searchRequest = searchRequestFactory.extendWithSavedIdentifiers(searchRequest);
searchRequest = customQueryAndTitleMapping.mapSearchRequest(searchRequest);
searchRequest = customQueryAndTitleMappingHandler.mapSearchRequest(searchRequest);
return searchRequest;
}
@Data
@ReflectionMarker
@AllArgsConstructor
private static class CacheEntryValue {
private final NewznabParameters params;

2
core/src/main/java/org/nzbhydra/api/NewznabJsonTransformer.java
View File

@ -17,6 +17,7 @@
package org.nzbhydra.api;
import org.nzbhydra.config.ConfigProvider;
import org.nzbhydra.config.downloading.DownloadType;
import org.nzbhydra.downloading.FileHandler;
import org.nzbhydra.mapping.newznab.NewznabResponse;
import org.nzbhydra.mapping.newznab.json.NewznabJsonChannel;
@ -29,7 +30,6 @@ import org.nzbhydra.mapping.newznab.json.NewznabJsonItemAttributes;
import org.nzbhydra.mapping.newznab.json.NewznabJsonResponseAttributes;
import org.nzbhydra.mapping.newznab.json.NewznabJsonRoot;
import org.nzbhydra.searching.dtoseventsenums.SearchResultItem;
import org.nzbhydra.searching.dtoseventsenums.SearchResultItem.DownloadType;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

2
core/src/main/java/org/nzbhydra/api/NewznabXmlTransformer.java
View File

@ -17,6 +17,7 @@
package org.nzbhydra.api;
import org.nzbhydra.config.ConfigProvider;
import org.nzbhydra.config.downloading.DownloadType;
import org.nzbhydra.downloading.FileHandler;
import org.nzbhydra.mapping.newznab.NewznabResponse;
import org.nzbhydra.mapping.newznab.xml.NewznabAttribute;
@ -27,7 +28,6 @@ import org.nzbhydra.mapping.newznab.xml.NewznabXmlItem;
import org.nzbhydra.mapping.newznab.xml.NewznabXmlResponse;
import org.nzbhydra.mapping.newznab.xml.NewznabXmlRoot;
import org.nzbhydra.searching.dtoseventsenums.SearchResultItem;
import org.nzbhydra.searching.dtoseventsenums.SearchResultItem.DownloadType;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

2
core/src/main/java/org/nzbhydra/api/stats/ApiStatsRequest.java
View File

@ -18,8 +18,10 @@ package org.nzbhydra.api.stats;
import lombok.Data;
import org.nzbhydra.historystats.stats.StatsRequest;
import org.nzbhydra.springnative.ReflectionMarker;
@Data
@ReflectionMarker
public class ApiStatsRequest {
protected String apikey;

8
core/src/main/java/org/nzbhydra/auth/AsyncSupportFilter.java
View File

@ -17,14 +17,14 @@
package org.nzbhydra.auth;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.apache.catalina.Globals;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;
import javax.servlet.FilterChain;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
@Component

8
core/src/main/java/org/nzbhydra/auth/AuthAndAccessEventHandler.java
View File

@ -1,5 +1,8 @@
package org.nzbhydra.auth;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.nzbhydra.notifications.AuthFailureNotificationEvent;
import org.nzbhydra.web.SessionStorage;
import org.slf4j.Logger;
@ -17,9 +20,6 @@ import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseStatus;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
@Controller
@ -69,7 +69,7 @@ public class AuthAndAccessEventHandler extends AccessDeniedHandlerImpl {
@Override
public void handle(HttpServletRequest request, HttpServletResponse response, AccessDeniedException accessDeniedException) throws IOException, ServletException {
logger.warn("Access denied to IP {}: {}", SessionStorage.IP.get(), accessDeniedException.getMessage());
logger.warn("Access denied to IP {}: {}. Request path: {}. Parameters: {}", SessionStorage.IP.get(), accessDeniedException.getMessage(), request.getContextPath(), request.getParameterMap());
attemptService.accessFailed(SessionStorage.IP.get());
super.handle(request, response, accessDeniedException);
}

2
core/src/main/java/org/nzbhydra/auth/AuthWeb.java
View File

@ -1,5 +1,6 @@
package org.nzbhydra.auth;
import jakarta.servlet.http.HttpSession;
import org.nzbhydra.web.BootstrappedDataTO;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.MediaType;
@ -10,7 +11,6 @@ import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;
import javax.servlet.http.HttpSession;
import java.security.Principal;
@RestController

8
core/src/main/java/org/nzbhydra/auth/ForwardedForRecognizingFilter.java
View File

@ -16,15 +16,15 @@
package org.nzbhydra.auth;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.nzbhydra.web.SessionStorage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.filter.OncePerRequestFilter;
import javax.servlet.FilterChain;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
public class ForwardedForRecognizingFilter extends OncePerRequestFilter {

33
core/src/main/java/org/nzbhydra/auth/HeaderAuthenticationFilter.java
View File

@ -17,24 +17,27 @@
package org.nzbhydra.auth;
import com.google.common.net.InetAddresses;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.nzbhydra.config.auth.AuthConfig;
import org.nzbhydra.web.SessionStorage;
import org.nzbhydra.webaccess.HydraOkHttp3ClientHttpRequestFactory;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.security.authentication.AnonymousAuthenticationToken;
import org.springframework.security.authentication.AuthenticationManager;
import org.springframework.security.authentication.BadCredentialsException;
import org.springframework.security.authentication.UsernamePasswordAuthenticationToken;
import org.springframework.security.core.authority.AuthorityUtils;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.security.core.userdetails.UserDetails;
import org.springframework.security.core.userdetails.UsernameNotFoundException;
import org.springframework.security.web.authentication.www.BasicAuthenticationFilter;
import javax.servlet.FilterChain;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.util.Objects;
public class HeaderAuthenticationFilter extends BasicAuthenticationFilter {
@ -43,14 +46,36 @@ public class HeaderAuthenticationFilter extends BasicAuthenticationFilter {
private final HydraUserDetailsManager userDetailsManager;
private AuthConfig authConfig;
private final String internalApiKey;
public HeaderAuthenticationFilter(AuthenticationManager authenticationManager, HydraUserDetailsManager userDetailsManager, AuthConfig authConfig) {
super(authenticationManager);
this.userDetailsManager = userDetailsManager;
this.authConfig = authConfig;
//Must be provided by wrapper
internalApiKey = System.getProperty("internalApiKey");
if (internalApiKey != null) {
logger.info("Using internal API key");
}
}
@Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain chain) throws IOException, ServletException {
final String sentInternalApiKey = request.getParameterValues("internalApiKey") == null ? null : request.getParameterValues("internalApiKey")[0];
if (sentInternalApiKey != null) {
if (Objects.equals(sentInternalApiKey, internalApiKey)) {
final AnonymousAuthenticationToken token = new AnonymousAuthenticationToken("key", "internalApi", AuthorityUtils.createAuthorityList("ROLE_ADMIN"));
token.setDetails(new HydraWebAuthenticationDetails(request));
SecurityContextHolder.getContext().setAuthentication(token);
onSuccessfulAuthentication(request, response, token);
logger.debug("Authorized access to {} via internal API key", request.getRequestURI());
chain.doFilter(request, response);
return;
} else {
logger.warn("Invalid internal API key provided");
}
}
if (authConfig.getAuthHeader() == null || authConfig.getAuthHeaderIpRanges().isEmpty()) {
chain.doFilter(request, response);
return;

16
core/src/main/java/org/nzbhydra/auth/HydraAnonymousAuthenticationFilter.java
View File

@ -1,7 +1,12 @@
package org.nzbhydra.auth;
import org.nzbhydra.config.BaseConfig;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.ServletRequest;
import jakarta.servlet.ServletResponse;
import jakarta.servlet.http.HttpServletRequest;
import org.nzbhydra.config.ConfigChangedEvent;
import org.nzbhydra.config.ConfigProvider;
import org.nzbhydra.config.auth.AuthConfig;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@ -18,11 +23,6 @@ import org.springframework.security.web.authentication.WebAuthenticationDetailsS
import org.springframework.stereotype.Component;
import org.springframework.util.Assert;
import javax.servlet.FilterChain;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
@ -43,9 +43,9 @@ public class HydraAnonymousAuthenticationFilter extends AnonymousAuthenticationF
//Disabled by default because just by existing it will be used for static resource accesses where spring security is disabled
private boolean enabled = false;
public HydraAnonymousAuthenticationFilter(@Autowired BaseConfig baseConfig) {
public HydraAnonymousAuthenticationFilter(@Autowired ConfigProvider configProvider) {
super("anonymous", "anonymousUser", AuthorityUtils.createAuthorityList("ROLE_ANONYMOUS"));
updateAuthorities(baseConfig.getAuth());
updateAuthorities(configProvider.getBaseConfig().getAuth());
}
public void enable() {

5
core/src/main/java/org/nzbhydra/auth/HydraEmbeddedServletContainer.java
View File

@ -16,6 +16,7 @@
package org.nzbhydra.auth;
import jakarta.servlet.ServletException;
import org.apache.catalina.connector.Request;
import org.apache.catalina.connector.Response;
import org.apache.catalina.valves.ValveBase;
@ -27,7 +28,6 @@ import org.springframework.boot.web.server.WebServerFactoryCustomizer;
import org.springframework.boot.web.servlet.server.ConfigurableServletWebServerFactory;
import org.springframework.stereotype.Component;
import javax.servlet.ServletException;
import java.io.IOException;
/**
@ -41,10 +41,9 @@ public class HydraEmbeddedServletContainer implements WebServerFactoryCustomizer
@Override
public void customize(ConfigurableServletWebServerFactory factory) {
if (!(factory instanceof TomcatServletWebServerFactory)) {
if (!(factory instanceof TomcatServletWebServerFactory containerFactory)) {
return; //Is the case in tests
}
TomcatServletWebServerFactory containerFactory = (TomcatServletWebServerFactory) factory;
containerFactory.addContextValves(new ValveBase() {
@Override
public void invoke(Request request, Response response) throws IOException, ServletException {

8
core/src/main/java/org/nzbhydra/auth/HydraGlobalMethodSecurityConfiguration.java
View File

@ -16,7 +16,7 @@
package org.nzbhydra.auth;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.config.ConfigProvider;
import org.nzbhydra.config.auth.AuthType;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@ -40,14 +40,14 @@ public class HydraGlobalMethodSecurityConfiguration extends GlobalMethodSecurity
private static final Logger hydraLogger = LoggerFactory.getLogger(HydraGlobalMethodSecurityConfiguration.class);
@Autowired
private BaseConfig baseConfig;
private ConfigProvider configProvider;
@Bean
public MethodSecurityMetadataSource methodSecurityMetadataSource() {
List<MethodSecurityMetadataSource> sources = new ArrayList<>();
if (baseConfig.getAuth().getAuthType() != AuthType.NONE) {
hydraLogger.info("Enabling auth type " + baseConfig.getAuth().getAuthType());
if (configProvider.getBaseConfig().getAuth().getAuthType() != AuthType.NONE) {
hydraLogger.info("Enabling auth type " + configProvider.getBaseConfig().getAuth().getAuthType());
sources.add(new SecuredAnnotationSecurityMetadataSource());
}

6
core/src/main/java/org/nzbhydra/auth/HydraWebAuthenticationDetails.java
View File

@ -1,10 +1,9 @@
package org.nzbhydra.auth;
import com.google.common.base.Objects;
import jakarta.servlet.http.HttpServletRequest;
import org.springframework.security.web.authentication.WebAuthenticationDetails;
import javax.servlet.http.HttpServletRequest;
public class HydraWebAuthenticationDetails extends WebAuthenticationDetails {
private final String filteredIp;
@ -39,10 +38,9 @@ public class HydraWebAuthenticationDetails extends WebAuthenticationDetails {
if (this == o) {
return true;
}
if (!(o instanceof HydraWebAuthenticationDetails)) {
if (!(o instanceof HydraWebAuthenticationDetails that)) {
return false;
}
HydraWebAuthenticationDetails that = (HydraWebAuthenticationDetails) o;
return Objects.equal(filteredIp, that.filteredIp);
}

2
core/src/main/java/org/nzbhydra/auth/LoginAndAccessAttemptService.java
View File

@ -19,7 +19,7 @@ public class LoginAndAccessAttemptService {
private final LoadingCache<String, Integer> attemptsCache;
public LoginAndAccessAttemptService() {
attemptsCache = CacheBuilder.newBuilder().expireAfterWrite(1, TimeUnit.DAYS).build(new CacheLoader<String, Integer>() {
attemptsCache = CacheBuilder.newBuilder().expireAfterWrite(1, TimeUnit.DAYS).build(new CacheLoader<>() {
@Override
public Integer load(String key) throws Exception {
return 0;

14
core/src/main/java/org/nzbhydra/auth/PersistentLoginsEntity.java
View File

@ -1,12 +1,13 @@
package org.nzbhydra.auth;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import jakarta.validation.constraints.NotNull;
import lombok.Data;
import org.nzbhydra.springnative.ReflectionMarker;
import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.Id;
import javax.persistence.Table;
import javax.validation.constraints.NotNull;
import java.sql.Timestamp;
/**
@ -16,7 +17,8 @@ import java.sql.Timestamp;
@Entity
@Table(name = "persistent_logins")
@Data
public class PersistentLoginsEntity {
@ReflectionMarker
public final class PersistentLoginsEntity {
@Id
@NotNull

119
core/src/main/java/org/nzbhydra/auth/SecurityConfig.java
View File

@ -1,5 +1,6 @@
package org.nzbhydra.auth;
import jakarta.servlet.http.HttpServletRequest;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.config.ConfigChangedEvent;
import org.nzbhydra.config.ConfigProvider;
@ -12,26 +13,25 @@ import org.springframework.context.annotation.Configuration;
import org.springframework.context.event.EventListener;
import org.springframework.core.annotation.Order;
import org.springframework.security.authentication.AuthenticationManager;
import org.springframework.security.config.BeanIds;
import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.builders.WebSecurity;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;
import org.springframework.security.core.userdetails.UserDetailsService;
import org.springframework.security.web.SecurityFilterChain;
import org.springframework.security.web.access.channel.ChannelProcessingFilter;
import org.springframework.security.web.authentication.WebAuthenticationDetails;
import org.springframework.security.web.authentication.WebAuthenticationDetailsSource;
import org.springframework.security.web.authentication.www.BasicAuthenticationFilter;
import org.springframework.security.web.csrf.CookieCsrfTokenRepository;
import org.springframework.security.web.csrf.CsrfTokenRequestAttributeHandler;
import org.springframework.security.web.firewall.DefaultHttpFirewall;
import org.springframework.security.web.util.matcher.AntPathRequestMatcher;
import org.springframework.web.filter.ForwardedHeaderFilter;
import javax.servlet.http.HttpServletRequest;
@Configuration
@Order
@EnableWebSecurity
public class SecurityConfig extends WebSecurityConfigurerAdapter {
public class SecurityConfig {
private static final Logger logger = LoggerFactory.getLogger(SecurityConfig.class);
private static final int SECONDS_PER_DAY = 60 * 60 * 24;
@ -45,27 +45,34 @@ public class SecurityConfig extends WebSecurityConfigurerAdapter {
@Autowired
private AuthAndAccessEventHandler authAndAccessEventHandler;
@Autowired
private UserDetailsService userDetailsService;
@Autowired
private AsyncSupportFilter asyncSupportFilter;
private HeaderAuthenticationFilter headerAuthenticationFilter;
@Override
protected void configure(HttpSecurity http) throws Exception {
@Bean
public SecurityFilterChain filterChain(HttpSecurity http, AuthenticationManager authenticationManager) throws Exception {
BaseConfig baseConfig = configProvider.getBaseConfig();
if (configProvider.getBaseConfig().getMain().isUseCsrf()) {
boolean useCsrf = Boolean.parseBoolean(System.getProperty("main.useCsrf"));
if (configProvider.getBaseConfig().getMain().isUseCsrf() && useCsrf) {
CookieCsrfTokenRepository csrfTokenRepository = CookieCsrfTokenRepository.withHttpOnlyFalse();
csrfTokenRepository.setCookieName("HYDRA-XSRF-TOKEN");
http.csrf().csrfTokenRepository(csrfTokenRepository);
//https://docs.spring.io/spring-security/reference/5.8/migration/servlet/exploits.html#_i_am_using_angularjs_or_another_javascript_framework
CsrfTokenRequestAttributeHandler requestHandler = new CsrfTokenRequestAttributeHandler();
requestHandler.setCsrfRequestAttributeName(null);
http.csrf()
.csrfTokenRepository(csrfTokenRepository)
.csrfTokenRequestHandler(requestHandler);
} else {
logger.info("CSRF is disabled");
http.csrf().disable();
}
http.headers().httpStrictTransportSecurity().disable();
http.headers().frameOptions().disable();
http.headers()
.httpStrictTransportSecurity().disable()
.frameOptions().disable();
if (baseConfig.getAuth().getAuthType() == AuthType.BASIC) {
http = http
.authorizeRequests()
.antMatchers("/internalapi/userinfos").permitAll()
.and()
.httpBasic()
.authenticationDetailsSource(new WebAuthenticationDetailsSource() {
@Override
@ -73,27 +80,48 @@ public class SecurityConfig extends WebSecurityConfigurerAdapter {
return new HydraWebAuthenticationDetails(context);
}
})
.and()
.logout().logoutUrl("/logout").deleteCookies("remember-me")
.and();
} else if (baseConfig.getAuth().getAuthType() == AuthType.FORM) {
http = http
.authorizeRequests()
.antMatchers("/internalapi/userinfos").permitAll()
.and()
.formLogin().loginPage("/login").loginProcessingUrl("/login").defaultSuccessUrl("/").permitAll()
.formLogin()
.loginPage("/login")
.loginProcessingUrl("/login")
.defaultSuccessUrl("/")
.permitAll()
.authenticationDetailsSource(new WebAuthenticationDetailsSource() {
@Override
public WebAuthenticationDetails buildDetails(HttpServletRequest context) {
return new HydraWebAuthenticationDetails(context);
}
})
.and()
.logout().permitAll().logoutUrl("/logout").logoutSuccessUrl("/loggedout").logoutSuccessUrl("/").deleteCookies("remember-me")
.and();
}
if (baseConfig.getAuth().isAuthConfigured()) {
http = http
.authorizeHttpRequests()
.requestMatchers("/internalapi/")
.authenticated()
.requestMatchers("/websocket/")
.authenticated()
.requestMatchers("/actuator/**")
.hasRole("ADMIN")
.requestMatchers(new AntPathRequestMatcher("/static/**"))
.permitAll()
.anyRequest()
// .authenticated() //Does not include anonymous
.hasAnyRole("ADMIN", "ANONYMOUS", "USER")
.and()
.logout()
.permitAll()
.logoutUrl("/logout")
.logoutSuccessUrl("/")
.deleteCookies("remember-me")
.invalidateHttpSession(true)
.clearAuthentication(true)
.and()
;
enableAnonymousAccessIfConfigured(http);
if (baseConfig.getAuth().isRememberUsers()) {
int rememberMeValidityDays = configProvider.getBaseConfig().getAuth().getRememberMeValidityDays();
if (rememberMeValidityDays == 0) {
@ -103,37 +131,36 @@ public class SecurityConfig extends WebSecurityConfigurerAdapter {
.rememberMe()
.alwaysRemember(true)
.tokenValiditySeconds(rememberMeValidityDays * SECONDS_PER_DAY)
.userDetailsService(userDetailsService())
.userDetailsService(userDetailsService)
.and();
}
http.authorizeRequests()
.antMatchers("/actuator/**")
.hasRole("ADMIN")
.anyRequest().permitAll();
}
headerAuthenticationFilter = new HeaderAuthenticationFilter(authenticationManager(), hydraUserDetailsManager, configProvider.getBaseConfig().getAuth());
http.addFilterBefore(new ForwardedForRecognizingFilter(), ChannelProcessingFilter.class);
//We need to extract the original IP before it's removed and not retrievable anymore by the ForwardedHeaderFilter
http.addFilterAfter(new ForwardedHeaderFilter(), ForwardedForRecognizingFilter.class);
headerAuthenticationFilter = new HeaderAuthenticationFilter(authenticationManager, hydraUserDetailsManager, configProvider.getBaseConfig().getAuth());
http.addFilterAfter(headerAuthenticationFilter, BasicAuthenticationFilter.class);
http.addFilterAfter(asyncSupportFilter, BasicAuthenticationFilter.class);
} else {
http.authorizeHttpRequests().anyRequest().permitAll();
}
http.exceptionHandling().accessDeniedHandler(authAndAccessEventHandler);
http.addFilterBefore(new ForwardedForRecognizingFilter(), ChannelProcessingFilter.class);
//We need to extract the original IP before it's removed and not retrievable anymore by the ForwardedHeaderFilter
http.addFilterAfter(new ForwardedHeaderFilter(), ForwardedForRecognizingFilter.class);
return http.build();
}
@Override
public void configure(WebSecurity web) {
web.ignoring().antMatchers("/static/**");
}
private void enableAnonymousAccessIfConfigured(HttpSecurity http) {
//Create an anonymous auth filter. If any of the areas are not restricted the anonymous user will get its role
try {
if (!hydraAnonymousAuthenticationFilter.getAuthorities().isEmpty()) {
http.anonymous().authenticationFilter(hydraAnonymousAuthenticationFilter);
hydraAnonymousAuthenticationFilter.enable();
}
} catch (Exception e) {
logger.error("Unable to configure anonymous access", e);
}
@ -146,21 +173,19 @@ public class SecurityConfig extends WebSecurityConfigurerAdapter {
}
}
@Bean(name = BeanIds.AUTHENTICATION_MANAGER)
@Override
public AuthenticationManager authenticationManagerBean() throws Exception {
return super.authenticationManagerBean();
}
@Bean
public DefaultHttpFirewall defaultHttpFirewall() {
//Allow duplicate trailing slashes which happen when behind a reverse proxy, e.g. proxy_pass http://127.0.0.1:5076/nzbhydra2/;
return new DefaultHttpFirewall();
}
@Override
public void configure(AuthenticationManagerBuilder auth) throws Exception {
auth.userDetailsService(hydraUserDetailsManager);
@Bean
public AuthenticationManager authManager(HttpSecurity http)
throws Exception {
return http.getSharedObject(AuthenticationManagerBuilder.class)
.userDetailsService(hydraUserDetailsManager)
.and()
.build();
}

77
core/src/main/java/org/nzbhydra/backup/BackupAndRestore.java
View File

@ -1,26 +1,28 @@
package org.nzbhydra.backup;
import com.google.common.base.Stopwatch;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import com.google.common.base.Throwables;
import jakarta.annotation.PostConstruct;
import jakarta.persistence.EntityManager;
import jakarta.persistence.PersistenceContext;
import jakarta.persistence.Query;
import org.apache.commons.io.FileUtils;
import org.h2.message.DbException;
import org.nzbhydra.GenericResponse;
import org.nzbhydra.NzbHydra;
import org.nzbhydra.config.ConfigProvider;
import org.nzbhydra.config.ConfigReaderWriter;
import org.nzbhydra.genericstorage.GenericStorage;
import org.nzbhydra.logging.LoggingMarkers;
import org.nzbhydra.systemcontrol.SystemControl;
import org.nzbhydra.webaccess.HydraOkHttp3ClientHttpRequestFactory;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.aot.hint.annotation.Reflective;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import org.springframework.transaction.annotation.Transactional;
import javax.annotation.PostConstruct;
import javax.persistence.EntityManager;
import javax.persistence.PersistenceContext;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
@ -36,7 +38,6 @@ import java.nio.file.StandardCopyOption;
import java.nio.file.StandardOpenOption;
import java.nio.file.attribute.BasicFileAttributes;
import java.nio.file.spi.FileSystemProvider;
import java.time.Instant;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
@ -64,6 +65,9 @@ public class BackupAndRestore {
private HydraOkHttp3ClientHttpRequestFactory requestFactory;
@Autowired
private SystemControl systemControl;
@Autowired
private GenericStorage genericStorage;
private final ConfigReaderWriter configReaderWriter = new ConfigReaderWriter();
@PostConstruct
@ -77,7 +81,7 @@ public class BackupAndRestore {
}
@Transactional
public File backup() throws Exception {
public File backup(boolean triggeredByUsed) throws Exception {
Stopwatch stopwatch = Stopwatch.createStarted();
File backupFolder = getBackupFolder();
if (!backupFolder.exists()) {
@ -92,7 +96,10 @@ public class BackupAndRestore {
logger.info("Creating backup");
File backupZip = new File(backupFolder, "nzbhydra-" + LocalDateTime.now().format(DATE_PATTERN) + ".zip");
backupDatabase(backupZip);
backupDatabase(backupZip, triggeredByUsed);
if (!backupZip.exists()) {
throw new RuntimeException("Export to file " + backupZip + " was not executed by database");
}
Map<String, String> env = new HashMap<>();
env.put("create", "true");
//We use the jar filesystem so we can add files to the existing ZIP
@ -117,8 +124,8 @@ public class BackupAndRestore {
logger.info("Deleting old backups if any exist");
Integer backupMaxAgeInWeeks = configProvider.getBaseConfig().getMain().getDeleteBackupsAfterWeeks().get();
File[] zips = backupFolder.listFiles((dir, name) -> name != null && name.startsWith("nzbhydra") && name.endsWith(".zip"));
if (zips != null) {
Map<File, LocalDateTime> fileToBackupTime = new HashMap<>();
for (File zip : zips) {
Matcher matcher = FILE_PATTERN.matcher(zip.getName());
if (!matcher.matches()) {
@ -126,8 +133,20 @@ public class BackupAndRestore {
continue;
}
LocalDateTime backupDate = LocalDateTime.from(DATE_PATTERN.parse(matcher.group(1)));
if (backupDate.isBefore(LocalDateTime.now().minusSeconds(60 * 60 * 24 * 7 * backupMaxAgeInWeeks))) {
logger.info("Deleting backup file {} because it's older than {} weeks", zip, backupMaxAgeInWeeks);
fileToBackupTime.put(zip, backupDate);
}
for (File zip : zips) {
if (!fileToBackupTime.containsKey(zip)) {
continue;
}
LocalDateTime backupDate = fileToBackupTime.get(zip);
if (backupDate.isBefore(LocalDateTime.now().minusSeconds(60L * 60 * 24 * 7 * backupMaxAgeInWeeks))) {
final boolean successfulNewerBackup = fileToBackupTime.entrySet().stream().anyMatch(x -> x.getKey() != zip && x.getValue().isAfter(backupDate));
if (!successfulNewerBackup) {
logger.warn("No successful backup was made after the creation of {}. Will not delete it.", zip.getAbsolutePath());
continue;
}
logger.info("Deleting backup file {} because it's older than {} weeks and a newer successful backup exists", zip, backupMaxAgeInWeeks);
boolean deleted = zip.delete();
if (!deleted) {
logger.warn("Unable to delete old backup file {}", zip.getName());
@ -138,6 +157,7 @@ public class BackupAndRestore {
}
}
@Reflective
protected File getBackupFolder() {
final String backupFolder = configProvider.getBaseConfig().getMain().getBackupFolder();
if (backupFolder.contains(File.separator)) {
@ -164,11 +184,31 @@ public class BackupAndRestore {
}
private void backupDatabase(File targetFile) {
private void backupDatabase(File targetFile, boolean triggeredByUsed) {
final String tempPath;
try {
tempPath = Files.createTempFile("nzbhydra", "zip").toFile().getAbsolutePath().replace("\\", "/");
} catch (IOException e) {
throw new RuntimeException(e);
}
try {
String formattedFilepath = targetFile.getAbsolutePath().replace("\\", "/");
logger.info("Backing up database to " + formattedFilepath);
//Write a script to ensure that the backed up database is actually valid
final Query nativeQuery = entityManager.createNativeQuery("SCRIPT TO '%s';".formatted(tempPath));
//If the database is corrupted this command will back it up without exception
entityManager.createNativeQuery("BACKUP TO '" + formattedFilepath + "';").executeUpdate();
logger.debug("Wrote database backup files to {}", targetFile.getAbsolutePath());
final List resultList = nativeQuery.getResultList();
logger.debug("Wrote database backup data to {}", targetFile.getAbsolutePath());
} catch (Exception e) {
logger.info("Deleting invalid backup file {}", targetFile);
targetFile.delete();
if (!triggeredByUsed) {
final String dbExceptionMessage = Throwables.getCausalChain(e).stream().filter(x -> x instanceof DbException).findFirst().map(Throwable::getMessage).orElse(null);
genericStorage.save("FAILED_BACKUP", new FailedBackupData(dbExceptionMessage));
}
throw e;
}
}
private void backupCertificates(FileSystem fileSystem) throws IOException {
@ -205,8 +245,8 @@ public class BackupAndRestore {
try {
File tempFile = File.createTempFile("nzbhydra-restore", ".zip");
FileUtils.copyInputStreamToFile(inputStream, tempFile);
restoreFromFile(tempFile);
tempFile.deleteOnExit();
restoreFromFile(tempFile);
return GenericResponse.ok();
} catch (Exception e) {
logger.error("Error while restoring", e);
@ -265,12 +305,5 @@ public class BackupAndRestore {
}
}
@Data
@AllArgsConstructor
@NoArgsConstructor
public static class BackupEntry {
private String filename;
private Instant creationDate;
}
}

11
core/src/main/java/org/nzbhydra/backup/BackupData.java
View File

@ -1,11 +1,12 @@
package org.nzbhydra.backup;
import lombok.Data;
import org.nzbhydra.springnative.ReflectionMarker;
import java.io.Serializable;
import java.time.LocalDateTime;
@Data
@ReflectionMarker
public class BackupData implements Serializable {
protected LocalDateTime lastBackup;
@ -18,5 +19,11 @@ public class BackupData implements Serializable {
this.lastBackup = LocalDateTime.now();
}
public LocalDateTime getLastBackup() {
return lastBackup;
}
public void setLastBackup(LocalDateTime lastBackup) {
this.lastBackup = lastBackup;
}
}

6
core/src/main/java/org/nzbhydra/backup/BackupTask.java
View File

@ -39,7 +39,7 @@ public class BackupTask {
int backupEveryXDays = configProvider.getBaseConfig().getMain().getBackupEveryXDays().get();
Optional<LocalDateTime> firstStartOptional = genericStorage.get("FirstStart", LocalDateTime.class);
if (!firstStartOptional.isPresent()) {
if (firstStartOptional.isEmpty()) {
logger.debug("First start date time not set (for some reason), aborting backup");
return;
}
@ -52,7 +52,7 @@ public class BackupTask {
Optional<BackupData> backupData = genericStorage.get(KEY, BackupData.class);
if (!backupData.isPresent()) {
if (backupData.isEmpty()) {
logger.debug("Executing first backup: {} days since first start and backup is to be executed every {} days", daysSinceFirstStart, backupEveryXDays);
executeBackup();
return;
@ -68,7 +68,7 @@ public class BackupTask {
private void executeBackup() {
try {
logger.info("Starting weekly backup");
backupAndRestore.backup();
backupAndRestore.backup(false);
genericStorage.save(KEY, new BackupData(LocalDateTime.now(clock)));
} catch (Exception e) {
logger.error("An error occured while doing a background backup", e);

5
core/src/main/java/org/nzbhydra/backup/BackupWeb.java
View File

@ -1,7 +1,6 @@
package org.nzbhydra.backup;
import org.nzbhydra.GenericResponse;
import org.nzbhydra.backup.BackupAndRestore.BackupEntry;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
@ -33,7 +32,7 @@ public class BackupWeb {
@Transactional
public Object backupAndDownload() throws Exception {
try {
File backupFile = backup.backup();
File backupFile = backup.backup(true);
logger.debug("Sending contents of file {}", backupFile.getAbsolutePath());
return ResponseEntity
@ -52,7 +51,7 @@ public class BackupWeb {
@Transactional
public GenericResponse backupOnly() throws Exception {
try {
backup.backup();
backup.backup(true);
return GenericResponse.ok();
} catch (Exception e) {
logger.error("Error while creating backup", e);

45
core/src/main/java/org/nzbhydra/backup/FailedBackupData.java Normal file
View File

@ -0,0 +1,45 @@
package org.nzbhydra.backup;
import org.nzbhydra.springnative.ReflectionMarker;
import java.io.Serializable;
import java.time.LocalDateTime;
@ReflectionMarker
public class FailedBackupData implements Serializable {
private final LocalDateTime time = LocalDateTime.now();
private boolean shown;
private String message;
public FailedBackupData() {
}
public FailedBackupData(String message) {
this.message = message;
}
public boolean isShown() {
return shown;
}
public void setShown(boolean shown) {
this.shown = shown;
}
public String getMessage() {
return message;
}
public void setMessage(String message) {
this.message = message;
}
public LocalDateTime getTime() {
return time;
}
}

306
core/src/main/java/org/nzbhydra/config/BaseConfig.java
View File

@ -1,306 +0,0 @@
package org.nzbhydra.config;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.google.common.base.Joiner;
import lombok.AccessLevel;
import lombok.Data;
import lombok.EqualsAndHashCode;
import lombok.Getter;
import lombok.Setter;
import lombok.ToString;
import org.javers.core.metamodel.annotation.DiffIgnore;
import org.nzbhydra.NzbHydra;
import org.nzbhydra.ShutdownEvent;
import org.nzbhydra.config.auth.AuthConfig;
import org.nzbhydra.config.category.CategoriesConfig;
import org.nzbhydra.config.downloading.DownloadingConfig;
import org.nzbhydra.config.indexer.IndexerConfig;
import org.nzbhydra.logging.LoggingMarkers;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.ApplicationEventPublisher;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;
import javax.annotation.PostConstruct;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.Timer;
import java.util.TimerTask;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
import java.util.stream.Collectors;
@Component
@Data
@EqualsAndHashCode(exclude = {"applicationEventPublisher"}, callSuper = false)
public class BaseConfig extends ValidatingConfig<BaseConfig> {
private static final Logger logger = LoggerFactory.getLogger(BaseConfig.class);
public static boolean isProductive = true;
@Autowired
@Getter(AccessLevel.NONE)
@Setter(AccessLevel.NONE)
@JsonIgnore
private ApplicationEventPublisher applicationEventPublisher;
@JsonIgnore
private boolean initialized = false;
@JsonIgnore
private ConfigReaderWriter configReaderWriter = new ConfigReaderWriter();
private AuthConfig auth = new AuthConfig();
private CategoriesConfig categoriesConfig = new CategoriesConfig();
private DownloadingConfig downloading = new DownloadingConfig();
@DiffIgnore
private List<IndexerConfig> indexers = new ArrayList<>();
private MainConfig main = new MainConfig();
private SearchingConfig searching = new SearchingConfig();
private NotificationConfig notificationConfig = new NotificationConfig();
@DiffIgnore
private Map<String, String> genericStorage = new HashMap<>();
@JsonIgnore
@Getter(AccessLevel.NONE)
@Setter(AccessLevel.NONE)
private Lock saveLock = new ReentrantLock();
@JsonIgnore
@Getter(AccessLevel.NONE)
@Setter(AccessLevel.NONE)
@ToString.Exclude
private BaseConfig toSave;
@JsonIgnore
@Getter(AccessLevel.NONE)
@Setter(AccessLevel.NONE)
private TimerTask delayedSaveTimerTask;
public void replace(BaseConfig newConfig) {
replace(newConfig, true);
}
private void replace(BaseConfig newConfig, boolean fireConfigChangedEvent) {
BaseConfig oldBaseConfig = configReaderWriter.getCopy(this);
main = newConfig.getMain();
categoriesConfig = newConfig.getCategoriesConfig();
indexers = newConfig.getIndexers().stream().sorted(Comparator.comparing(IndexerConfig::getName)).collect(Collectors.toList());
downloading = newConfig.getDownloading();
searching = newConfig.getSearching();
auth = newConfig.getAuth();
genericStorage = newConfig.genericStorage;
notificationConfig = newConfig.notificationConfig;
if (fireConfigChangedEvent) {
ConfigChangedEvent configChangedEvent = new ConfigChangedEvent(this, oldBaseConfig, this);
applicationEventPublisher.publishEvent(configChangedEvent);
}
}
public void save(boolean saveInstantly) {
saveLock.lock();
if (saveInstantly) {
logger.debug(LoggingMarkers.CONFIG_READ_WRITE, "Saving instantly");
configReaderWriter.save(this);
toSave = null;
} else {
logger.debug(LoggingMarkers.CONFIG_READ_WRITE, "Delaying save");
toSave = this;
}
saveLock.unlock();
}
@PostConstruct
public void init() throws IOException {
if (!isProductive) {
//Don't overwrite settings from test code
initialized = true;
return;
}
if (initialized) {
//In some cases a call to the server will attempt to restart everything, trying to initialize beans. This
//method is called a second time and an empty / initial config is written
logger.warn("Init method called again. This can only happen during a faulty shutdown");
return;
}
logger.info("Using data folder {}", NzbHydra.getDataFolder());
replace(configReaderWriter.loadSavedConfig(), false);
if (main.getApiKey() == null) {
initializeNewConfig();
}
//Always save config to keep it in sync with base config (remove obsolete settings and add new ones)
configReaderWriter.save(this);
delayedSaveTimerTask = new TimerTask() {
@Override
public void run() {
saveToSave();
}
};
Timer delayedSaveTimer = new Timer("delayedConfigSave", false);
delayedSaveTimer.scheduleAtFixedRate(delayedSaveTimerTask, 10000, 10000);
initialized = true;
}
public void load() throws IOException {
replace(configReaderWriter.loadSavedConfig());
}
@EventListener
public void onShutdown(ShutdownEvent event) {
saveToSave();
}
private void saveToSave() {
saveLock.lock();
if (toSave != null) {
logger.debug(LoggingMarkers.CONFIG_READ_WRITE, "Executing delayed save");
configReaderWriter.save(toSave);
toSave = null;
}
saveLock.unlock();
}
@Override
public ConfigValidationResult validateConfig(BaseConfig oldConfig, BaseConfig newConfig, BaseConfig newBaseConfig) {
ConfigValidationResult configValidationResult = new ConfigValidationResult();
ConfigValidationResult authValidation = newConfig.getAuth().validateConfig(oldConfig, newConfig.getAuth(), newBaseConfig);
configValidationResult.getErrorMessages().addAll(authValidation.getErrorMessages());
configValidationResult.getWarningMessages().addAll(authValidation.getWarningMessages());
configValidationResult.setRestartNeeded(configValidationResult.isRestartNeeded() || authValidation.isRestartNeeded());
ConfigValidationResult categoriesValidation = newConfig.getCategoriesConfig().validateConfig(oldConfig, newConfig.getCategoriesConfig(), newBaseConfig);
configValidationResult.getErrorMessages().addAll(categoriesValidation.getErrorMessages());
configValidationResult.getWarningMessages().addAll(categoriesValidation.getWarningMessages());
configValidationResult.setRestartNeeded(configValidationResult.isRestartNeeded() || categoriesValidation.isRestartNeeded());
ConfigValidationResult mainValidation = newConfig.getMain().validateConfig(oldConfig, newConfig.getMain(), newBaseConfig);
configValidationResult.getErrorMessages().addAll(mainValidation.getErrorMessages());
configValidationResult.getWarningMessages().addAll(mainValidation.getWarningMessages());
configValidationResult.setRestartNeeded(configValidationResult.isRestartNeeded() || mainValidation.isRestartNeeded());
ConfigValidationResult searchingValidation = newConfig.getSearching().validateConfig(oldConfig, newConfig.getSearching(), newBaseConfig);
configValidationResult.getErrorMessages().addAll(searchingValidation.getErrorMessages());
configValidationResult.getWarningMessages().addAll(searchingValidation.getWarningMessages());
configValidationResult.setRestartNeeded(configValidationResult.isRestartNeeded() || searchingValidation.isRestartNeeded());
ConfigValidationResult downloadingValidation = newConfig.getDownloading().validateConfig(oldConfig, newConfig.getDownloading(), newBaseConfig);
configValidationResult.getErrorMessages().addAll(downloadingValidation.getErrorMessages());
configValidationResult.getWarningMessages().addAll(downloadingValidation.getWarningMessages());
configValidationResult.setRestartNeeded(configValidationResult.isRestartNeeded() || downloadingValidation.isRestartNeeded());
for (IndexerConfig indexer : newConfig.getIndexers()) {
ConfigValidationResult indexerValidation = indexer.validateConfig(oldConfig, indexer, newBaseConfig);
configValidationResult.getErrorMessages().addAll(indexerValidation.getErrorMessages());
configValidationResult.getWarningMessages().addAll(indexerValidation.getWarningMessages());
configValidationResult.setRestartNeeded(configValidationResult.isRestartNeeded() || indexerValidation.isRestartNeeded());
}
validateIndexers(newConfig, configValidationResult);
if (!configValidationResult.getErrorMessages().isEmpty()) {
logger.warn("Config validation returned errors:\n" + Joiner.on("\n").join(configValidationResult.getErrorMessages()));
}
if (!configValidationResult.getWarningMessages().isEmpty()) {
logger.warn("Config validation returned warnings:\n" + Joiner.on("\n").join(configValidationResult.getWarningMessages()));
}
if (configValidationResult.isRestartNeeded()) {
logger.warn("Settings were changed that require a restart to become effective");
}
configValidationResult.setOk(configValidationResult.getErrorMessages().isEmpty());
return configValidationResult;
}
private void validateIndexers(BaseConfig newConfig, ConfigValidationResult configValidationResult) {
if (!newConfig.getIndexers().isEmpty()) {
if (newConfig.getIndexers().stream().noneMatch(x -> x.getState() == IndexerConfig.State.ENABLED)) {
configValidationResult.getWarningMessages().add("No indexers enabled. Searches will return empty results");
} else if (newConfig.getIndexers().stream().allMatch(x -> x.getSupportedSearchIds().isEmpty())) {
if (newConfig.getSearching().getGenerateQueries() == SearchSourceRestriction.NONE) {
configValidationResult.getWarningMessages().add("No indexer found that supports search IDs. Without query generation searches using search IDs will return empty results.");
} else if (newConfig.getSearching().getGenerateQueries() != SearchSourceRestriction.BOTH) {
String name = newConfig.getSearching().getGenerateQueries() == SearchSourceRestriction.API ? "internal" : "API";
configValidationResult.getWarningMessages().add("No indexer found that supports search IDs. Without query generation " + name + " searches using search IDs will return empty results.");
}
}
Set<String> indexerNames = new HashSet<>();
Set<String> duplicateIndexerNames = new HashSet<>();
for (IndexerConfig indexer : newConfig.getIndexers()) {
if (!indexerNames.add(indexer.getName())) {
duplicateIndexerNames.add(indexer.getName());
}
}
if (!duplicateIndexerNames.isEmpty()) {
configValidationResult.getErrorMessages().add("Duplicate indexer names found: " + Joiner.on(", ").join(duplicateIndexerNames));
}
final Set<Set<String>> indexersSameHostAndApikey = new HashSet<>();
for (IndexerConfig indexer : newConfig.getIndexers()) {
final Set<String> otherIndexersSameHostAndApiKey = newConfig.getIndexers().stream()
.filter(x -> x != indexer)
.filter(x -> IndexerConfig.isIndexerEquals(x, indexer))
.map(IndexerConfig::getName)
.collect(Collectors.toSet());
if (!otherIndexersSameHostAndApiKey.isEmpty()) {
otherIndexersSameHostAndApiKey.add(indexer.getName());
if (indexersSameHostAndApikey.stream().noneMatch(x -> x.contains(indexer.getName()))) {
indexersSameHostAndApikey.add(otherIndexersSameHostAndApiKey);
final String message = "Found multiple indexers with same host and API key: " + Joiner.on(", ").join(otherIndexersSameHostAndApiKey);
logger.warn(message);
configValidationResult.getWarningMessages().add(message);
}
}
}
} else {
configValidationResult.getWarningMessages().add("No indexers configured. You won't get any results");
}
}
@Override
public BaseConfig prepareForSaving(BaseConfig oldBaseConfig) {
getCategoriesConfig().prepareForSaving(oldBaseConfig);
getDownloading().prepareForSaving(oldBaseConfig);
getSearching().prepareForSaving(oldBaseConfig);
getMain().prepareForSaving(oldBaseConfig);
getMain().getLogging().prepareForSaving(oldBaseConfig);
getAuth().prepareForSaving(oldBaseConfig);
getIndexers().forEach(indexerConfig -> indexerConfig.prepareForSaving(oldBaseConfig));
return this;
}
@Override
public BaseConfig updateAfterLoading() {
getAuth().updateAfterLoading();
return this;
}
@Override
public BaseConfig initializeNewConfig() {
getCategoriesConfig().initializeNewConfig();
getDownloading().initializeNewConfig();
getSearching().initializeNewConfig();
getMain().initializeNewConfig();
getAuth().initializeNewConfig();
return this;
}
}

147
core/src/main/java/org/nzbhydra/config/BaseConfigHandler.java Normal file
View File

@ -0,0 +1,147 @@
/*
* (C) Copyright 2023 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra.config;
import jakarta.annotation.PostConstruct;
import jakarta.annotation.PreDestroy;
import org.nzbhydra.NzbHydra;
import org.nzbhydra.config.indexer.IndexerConfig;
import org.nzbhydra.config.validation.BaseConfigValidator;
import org.nzbhydra.logging.LoggingMarkers;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.ApplicationEventPublisher;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.util.Comparator;
import java.util.Timer;
import java.util.TimerTask;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
import java.util.stream.Collectors;
@Component
public class BaseConfigHandler {
private static final Logger logger = LoggerFactory.getLogger(BaseConfigHandler.class);
private final ConfigReaderWriter configReaderWriter = new ConfigReaderWriter();
private final Lock saveLock = new ReentrantLock();
private BaseConfig toSave;
private TimerTask delayedSaveTimerTask;
public boolean initialized = false;
@Autowired
private ApplicationEventPublisher applicationEventPublisher;
@Autowired
private BaseConfig baseConfig;
@Autowired
private BaseConfigValidator baseConfigValidator;
@PostConstruct
public void init() throws IOException {
if (initialized) {
//In some cases a call to the server will attempt to restart everything, trying to initialize beans. This
//method is called a second time and an empty / initial config is written
logger.warn("Init method called again. This can only happen during a faulty shutdown");
return;
}
logger.info("Using data folder {}", NzbHydra.getDataFolder());
replace(configReaderWriter.loadSavedConfig(), false);
if (baseConfig.getMain().getApiKey() == null) {
baseConfigValidator.initializeNewConfig(baseConfig);
}
//Always save config to keep it in sync with base config (remove obsolete settings and add new ones)
configReaderWriter.save(baseConfig);
delayedSaveTimerTask = new TimerTask() {
@Override
public void run() {
saveToSave();
}
};
Timer delayedSaveTimer = new Timer("delayedConfigSave", false);
delayedSaveTimer.scheduleAtFixedRate(delayedSaveTimerTask, 10000, 10000);
initialized = true;
}
public void replace(BaseConfig newConfig) {
replace(newConfig, true);
}
public void replace(BaseConfig newConfig, boolean fireConfigChangedEvent) {
BaseConfig oldBaseConfig = configReaderWriter.getCopy(baseConfig);
baseConfig.setMain(newConfig.getMain());
baseConfig.setIndexers(newConfig.getIndexers().stream().sorted(Comparator.comparing(IndexerConfig::getName)).collect(Collectors.toList()));
baseConfig.setCategoriesConfig(newConfig.getCategoriesConfig());
baseConfig.setSearching(newConfig.getSearching());
baseConfig.setDownloading(newConfig.getDownloading());
baseConfig.setAuth(newConfig.getAuth());
baseConfig.setGenericStorage(newConfig.getGenericStorage());
baseConfig.setNotificationConfig(newConfig.getNotificationConfig());
if (fireConfigChangedEvent) {
ConfigChangedEvent configChangedEvent = new ConfigChangedEvent(this, oldBaseConfig, newConfig);
applicationEventPublisher.publishEvent(configChangedEvent);
}
}
public void save(boolean saveInstantly) {
saveLock.lock();
if (saveInstantly) {
logger.debug(LoggingMarkers.CONFIG_READ_WRITE, "Saving instantly");
configReaderWriter.save(baseConfig);
toSave = null;
} else {
logger.debug(LoggingMarkers.CONFIG_READ_WRITE, "Delaying save");
toSave = baseConfig;
}
saveLock.unlock();
}
public void load() throws IOException {
replace(configReaderWriter.loadSavedConfig());
}
@PreDestroy
public void onShutdown() {
saveToSave();
delayedSaveTimerTask.cancel();
}
private void saveToSave() {
saveLock.lock();
if (toSave != null) {
logger.debug(LoggingMarkers.CONFIG_READ_WRITE, "Executing delayed save");
configReaderWriter.save(toSave);
toSave = null;
}
saveLock.unlock();
}
}

3
core/src/main/java/org/nzbhydra/config/ConfigProvider.java
View File

@ -2,9 +2,12 @@ package org.nzbhydra.config;
import org.nzbhydra.config.indexer.IndexerConfig;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.DependsOn;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;
//BaseConfig must be initialized before we can provide it
@DependsOn("baseConfigHandler")
@Component
public class ConfigProvider {

30
core/src/main/java/org/nzbhydra/config/ConfigReaderWriter.java
View File

@ -21,8 +21,10 @@ import com.fasterxml.jackson.core.type.TypeReference;
import com.google.common.base.Charsets;
import com.google.common.base.Stopwatch;
import com.google.common.base.Strings;
import net.jodah.failsafe.Failsafe;
import net.jodah.failsafe.RetryPolicy;
import dev.failsafe.Failsafe;
import dev.failsafe.RetryPolicy;
import dev.failsafe.event.EventListener;
import dev.failsafe.event.ExecutionCompletedEvent;
import org.apache.commons.io.IOUtils;
import org.nzbhydra.Jackson;
import org.nzbhydra.NzbHydra;
@ -38,6 +40,7 @@ import java.io.InputStream;
import java.io.InputStreamReader;
import java.nio.file.Files;
import java.nio.file.StandardCopyOption;
import java.time.Duration;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.TimeUnit;
@ -47,9 +50,9 @@ public class ConfigReaderWriter {
private static final Logger logger = LoggerFactory.getLogger(ConfigReaderWriter.class);
public static final TypeReference<HashMap<String, Object>> MAP_TYPE_REFERENCE = new TypeReference<HashMap<String, Object>>() {
public static final TypeReference<HashMap<String, Object>> MAP_TYPE_REFERENCE = new TypeReference<>() {
};
private final RetryPolicy saveRetryPolicy = new RetryPolicy().retryOn(IOException.class).withDelay(1000, TimeUnit.MILLISECONDS).withMaxRetries(3);
private final RetryPolicy saveRetryPolicy = RetryPolicy.builder().withDelay(Duration.ofMillis(1000)).withMaxRetries(3).handle(IOException.class).build();
public void save(BaseConfig baseConfig) {
@ -81,10 +84,19 @@ public class ConfigReaderWriter {
save(converted, buildConfigFileFile());
}
@SuppressWarnings({"Convert2Lambda", "Convert2Diamond"}) // Will not work with diamond
protected void save(File targetFile, String configAsYamlString) {
if (NzbHydra.isNativeBuild()) {
return;
}
synchronized (Jackson.YAML_MAPPER) {
Failsafe.with(saveRetryPolicy)
.onFailure(throwable -> logger.error("Unable to save config", throwable))
.onFailure(new EventListener<ExecutionCompletedEvent<Object>>() {
@Override
public void accept(ExecutionCompletedEvent<Object> event) throws Throwable {
logger.error("Unable to save config", event.getException());
}
})
.run(() -> doWrite(targetFile, configAsYamlString))
;
}
@ -117,14 +129,17 @@ public class ConfigReaderWriter {
*
* @param yamlFile The path of the file to be created
* @return true if initialization was needed
* @throws IOException
*/
public boolean initializeIfNeeded(File yamlFile) throws IOException {
if (NzbHydra.isNativeBuild()) {
return false;
}
if (!yamlFile.exists()) {
logger.info("No config file found at {}. Initializing with base config", yamlFile);
try {
try (InputStream stream = BaseConfig.class.getResource("/config/baseConfig.yml").openStream()) {
logger.debug(LoggingMarkers.CONFIG_READ_WRITE, "Copying YAML to {}", yamlFile);
yamlFile.getParentFile().mkdirs();
Files.copy(stream, yamlFile.toPath());
return true;
}
@ -137,6 +152,9 @@ public class ConfigReaderWriter {
}
public void validateExistingConfig() {
if (NzbHydra.isNativeBuild()) {
return;
}
File configFile = buildConfigFileFile();
if (!configFile.exists()) {
logger.debug(LoggingMarkers.CONFIG_READ_WRITE, "Config file {} doesn't exist. Nothing to validate", configFile);

25
core/src/main/java/org/nzbhydra/config/ConfigWeb.java
View File

@ -1,16 +1,19 @@
package org.nzbhydra.config;
import jakarta.servlet.http.HttpSession;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import org.nzbhydra.GenericResponse;
import org.nzbhydra.config.FileSystemBrowser.DirectoryListingRequest;
import org.nzbhydra.config.FileSystemBrowser.FileSystemEntry;
import org.nzbhydra.config.ValidatingConfig.ConfigValidationResult;
import org.nzbhydra.config.indexer.IndexerConfig;
import org.nzbhydra.config.safeconfig.SafeConfig;
import org.nzbhydra.config.validation.BaseConfigValidator;
import org.nzbhydra.config.validation.ConfigValidationResult;
import org.nzbhydra.indexers.IndexerEntity;
import org.nzbhydra.indexers.IndexerRepository;
import org.nzbhydra.springnative.ReflectionMarker;
import org.nzbhydra.web.UrlCalculator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@ -25,7 +28,6 @@ import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.util.UriComponentsBuilder;
import javax.servlet.http.HttpSession;
import java.io.IOException;
import java.util.HashSet;
import java.util.LinkedHashMap;
@ -48,18 +50,22 @@ public class ConfigWeb {
private UrlCalculator urlCalculator;
@Autowired
private IndexerRepository indexerRepository;
@Autowired
private BaseConfigValidator baseConfigValidator;
@Autowired
private BaseConfigHandler baseConfigHandler;
private final ConfigReaderWriter configReaderWriter = new ConfigReaderWriter();
@Secured({"ROLE_ADMIN"})
@RequestMapping(value = "/internalapi/config", method = RequestMethod.GET, produces = MediaType.APPLICATION_JSON_VALUE)
public BaseConfig getConfig(HttpSession session) throws IOException {
return configReaderWriter.loadSavedConfig().updateAfterLoading();
final BaseConfig baseConfig = configReaderWriter.loadSavedConfig();
return baseConfigValidator.updateAfterLoading(baseConfig);
}
@Secured({"ROLE_ADMIN"})
@RequestMapping(value = "/internalapi/config", method = RequestMethod.PUT, consumes = MediaType.APPLICATION_JSON_VALUE, produces = MediaType.APPLICATION_JSON_VALUE)
public ConfigValidationResult setConfig(@RequestBody BaseConfig newConfig) throws IOException {
for (PropertySource<?> source : environment.getPropertySources()) {
Set propertyNames = new HashSet();
if (source.getSource() instanceof Properties) {
@ -74,13 +80,13 @@ public class ConfigWeb {
}
logger.info("Received new config");
newConfig = newConfig.prepareForSaving(configProvider.getBaseConfig());
ConfigValidationResult result = newConfig.validateConfig(configProvider.getBaseConfig(), newConfig, newConfig);
newConfig = baseConfigValidator.prepareForSaving(configProvider.getBaseConfig(), newConfig);
ConfigValidationResult result = baseConfigValidator.validateConfig(configProvider.getBaseConfig(), newConfig, newConfig);
if (result.isOk()) {
handleRenamedIndexers(newConfig);
configProvider.getBaseConfig().replace(newConfig);
configProvider.getBaseConfig().save(true);
baseConfigHandler.replace(newConfig);
baseConfigHandler.save(true);
result.setNewConfig(configProvider.getBaseConfig());
}
return result;
@ -120,7 +126,7 @@ public class ConfigWeb {
public GenericResponse reloadConfig() throws IOException {
logger.info("Reloading config from file");
try {
configProvider.getBaseConfig().load();
baseConfigHandler.load();
} catch (IOException e) {
return new GenericResponse(false, e.getMessage());
}
@ -150,6 +156,7 @@ public class ConfigWeb {
}
@Data
@ReflectionMarker
@AllArgsConstructor
@NoArgsConstructor
private static class ApiHelpResponse {

4
core/src/main/java/org/nzbhydra/config/FileSystemBrowser.java
View File

@ -3,6 +3,7 @@ package org.nzbhydra.config;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import org.nzbhydra.springnative.ReflectionMarker;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
@ -47,6 +48,7 @@ public class FileSystemBrowser {
}
@Data
@ReflectionMarker
@AllArgsConstructor
@NoArgsConstructor
public static class DirectoryListingRequest {
@ -56,6 +58,7 @@ public class FileSystemBrowser {
}
@Data
@ReflectionMarker
@AllArgsConstructor
@NoArgsConstructor
public static class FileSystemEntry {
@ -104,6 +107,7 @@ public class FileSystemBrowser {
}
@Data
@ReflectionMarker
@AllArgsConstructor
@NoArgsConstructor
public static class FileSystemSubEntry {

67
core/src/main/java/org/nzbhydra/config/LoggingConfig.java
View File

@ -1,67 +0,0 @@
package org.nzbhydra.config;
import lombok.Data;
import org.nzbhydra.logging.LoggingMarkers;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Iterator;
import java.util.List;
@Data
public class LoggingConfig extends ValidatingConfig<LoggingConfig> {
private static final Logger logger = LoggerFactory.getLogger(LoggingConfig.class);
@RestartRequired
private String consolelevel;
private HistoryUserInfoType historyUserInfoType = HistoryUserInfoType.NONE;
private boolean logIpAddresses;
private boolean mapIpToHost;
@RestartRequired
private boolean logGc;
private int logMaxHistory;
@RestartRequired
private String logfilelevel;
private boolean logUsername;
private List<String> markersToLog = new ArrayList<>();
@Override
public ConfigValidationResult validateConfig(BaseConfig oldConfig, LoggingConfig newLoggingConfig, BaseConfig newBaseConfig) {
ConfigValidationResult result = new ConfigValidationResult();
result.setRestartNeeded(isRestartNeeded(newBaseConfig.getMain().getLogging()));
if (newBaseConfig.getMain().getLogging().getMarkersToLog().size() > 3) {
result.getWarningMessages().add("You have more than 3 logging markers enabled. This is very rarely useful. Please make sure that this is actually needed. When creating debug infos please only enable those markers requested by the developer.");
}
return result;
}
@Override
public LoggingConfig prepareForSaving(BaseConfig oldBaseConfig) {
for (Iterator<String> iterator = markersToLog.iterator(); iterator.hasNext(); ) {
String marker = iterator.next();
if (Arrays.stream(LoggingMarkers.class.getDeclaredFields()).noneMatch(x -> x.getName().equals(marker))) {
logger.info("Removing logging marker that doesn't exist anymore.");
iterator.remove();
}
}
return this;
}
@Override
public LoggingConfig updateAfterLoading() {
return this;
}
@Override
public LoggingConfig initializeNewConfig() {
return this;
}
}

243
core/src/main/java/org/nzbhydra/config/MainConfig.java
View File

@ -1,243 +0,0 @@
package org.nzbhydra.config;
import com.fasterxml.jackson.annotation.JsonFormat;
import com.fasterxml.jackson.annotation.JsonFormat.Shape;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.google.common.base.Strings;
import lombok.Data;
import org.javers.core.metamodel.annotation.DiffIgnore;
import org.nzbhydra.NzbHydra;
import org.nzbhydra.config.downloading.ProxyType;
import org.nzbhydra.config.sensitive.SensitiveData;
import org.nzbhydra.debuginfos.DebugInfosProvider;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.context.properties.ConfigurationProperties;
import java.io.File;
import java.io.IOException;
import java.math.BigInteger;
import java.net.InetAddress;
import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
import java.util.Random;
@ConfigurationProperties("main")
@Data
public class MainConfig extends ValidatingConfig<MainConfig> {
private static final Logger logger = LoggerFactory.getLogger(MainConfig.class);
private Integer configVersion = 19;
//Hosting settings
@RestartRequired
private String host = "0.0.0.0";
@RestartRequired
private int port = 5076;
@RestartRequired
protected String urlBase = null;
//Proxy settings
@RestartRequired
@JsonFormat(shape = Shape.STRING)
private ProxyType proxyType = ProxyType.NONE;
@SensitiveData
private String proxyHost = null;
private int proxyPort;
private boolean proxyIgnoreLocal = true;
private List<String> proxyIgnoreDomains = new ArrayList<>();
@SensitiveData
private String proxyUsername;
@SensitiveData
private String proxyPassword;
//Database settings
private String backupFolder;
private Integer backupEveryXDays = 7;
private boolean backupBeforeUpdate = true;
private Integer deleteBackupsAfterWeeks = 4;
//History settings
private boolean keepHistory = true;
private Integer keepStatsForWeeks = null;
private Integer keepHistoryForWeeks = null;
//SSL settings
@RestartRequired
private boolean ssl = false;
@RestartRequired
private String sslKeyStore = null;
@SensitiveData
@RestartRequired
private String sslKeyStorePassword = null;
//Security settings
@RestartRequired
private boolean verifySsl = true;
private boolean disableSslLocally = false;
private List<String> sniDisabledFor = new ArrayList<>();
private List<String> verifySslDisabledFor = new ArrayList<>();
//Update settings
private boolean updateAutomatically = false;
private boolean updateToPrereleases = false;
private boolean updateCheckEnabled = true;
@JsonProperty("showUpdateBannerOnDocker")
private boolean showUpdateBannerOnUpdatedExternally = true;
private boolean showWhatsNewBanner = true;
//Startup / GUI settings
private boolean showNews = true;
private boolean startupBrowser = true;
private boolean welcomeShown = false;
protected String theme;
//Database settings
@RestartRequired
private int databaseCompactTime = 15_000;
@RestartRequired
private int databaseRetentionTime = 1000;
@RestartRequired
private int databaseWriteDelay = 5000;
//Other settings
@SensitiveData
@DiffIgnore
private String apiKey = null;
private String dereferer = null;
private boolean instanceCounterDownloaded = false;
private String repositoryBase;
private boolean shutdownForRestart = false;
@RestartRequired
private boolean useCsrf = true;
@RestartRequired
private int xmx;
private LoggingConfig logging = new LoggingConfig();
public Optional<String> getUrlBase() {
return Optional.ofNullable(Strings.emptyToNull(urlBase));
}
public Optional<Integer> getDeleteBackupsAfterWeeks() {
return Optional.ofNullable(deleteBackupsAfterWeeks);
}
public Optional<String> getDereferer() {
return Optional.ofNullable(dereferer); //This must be returned as empty string so that the config can overwrite it
}
public Optional<Integer> getBackupEveryXDays() {
return Optional.ofNullable(backupEveryXDays);
}
@Override
public ConfigValidationResult validateConfig(BaseConfig oldConfig, MainConfig newMainConfig, BaseConfig newBaseConfig) {
ConfigValidationResult result = new ConfigValidationResult();
MainConfig oldMain = oldConfig.getMain();
boolean portChanged = oldMain.getPort() != port;
boolean urlBaseChanged = oldMain.getUrlBase().isPresent() && !oldMain.getUrlBase().get().equals(urlBase);
if (urlBase == null && oldMain.getUrlBase().isPresent() && oldMain.getUrlBase().get().equals("/")) {
urlBaseChanged = false;
}
boolean sslChanged = oldMain.isSsl() != isSsl();
if (portChanged || urlBaseChanged || sslChanged && !startupBrowser) {
result.getWarningMessages().add("You've made changes that affect Hydra's URL and require a restart. Hydra will try and reload using the new URL when it's back.");
}
if (DebugInfosProvider.isRunInDocker() && !"0.0.0.0".equals(host)) {
result.getWarningMessages().add("You've changed the host but NZBHydra seems to be run in docker. It's recommended to use the host '0.0.0.0'.");
}
if (!"0.0.0.0".equals(host)) {
try {
boolean reachable = InetAddress.getByName(host).isReachable(1);
if (!reachable) {
result.getWarningMessages().add("The configured host address cannot be reached. Are you sure it is correct?");
}
} catch (IOException e) {
//Ignore, user will have to know what he does
}
}
if (oldConfig.getMain().getXmx() < 128) {
result.getErrorMessages().add("The JVM memory must be set to at least 128");
}
MainConfig newMain = newBaseConfig.getMain();
if (newMain.getKeepHistoryForWeeks() != null && newMain.getKeepHistoryForWeeks() <= 0) {
result.getErrorMessages().add("Please either delete the value for \"Keep history for\" or set it to a positive value.");
}
if (newMain.getKeepStatsForWeeks() != null && newMain.getKeepStatsForWeeks() <= 0) {
result.getErrorMessages().add("Please either delete the value for \"Keep stats for\" or set it to a positive value.");
}
if (newMain.getKeepStatsForWeeks() != null && newMain.getKeepHistoryForWeeks() != null && newMain.getKeepStatsForWeeks() > newMain.getKeepHistoryForWeeks()) {
result.getErrorMessages().add("Please set the time to keep stats to a value not higher than the time to keep history.");
}
if (newMain.getBackupFolder() != null) {
final File backupFolderFile;
if (backupFolder.contains(File.separator)) {
backupFolderFile = new File(backupFolder);
} else {
backupFolderFile = new File(NzbHydra.getDataFolder(), backupFolder);
}
if (!backupFolderFile.exists()) {
final boolean created = backupFolderFile.mkdirs();
if (!created) {
result.getErrorMessages().add("Backup folder " + backupFolder + " does not exist and could not be created");
}
}
}
ConfigValidationResult validationResult = getLogging().validateConfig(oldConfig, getLogging(), newBaseConfig);
result.getWarningMessages().addAll(validationResult.getWarningMessages());
result.getErrorMessages().addAll(validationResult.getErrorMessages());
oldMain = oldMain.prepareForSaving(oldConfig);
result.setRestartNeeded(validationResult.isRestartNeeded() || isRestartNeeded(oldMain));
result.setOk(validationResult.isOk() && result.isOk());
return result;
}
@Override
public MainConfig prepareForSaving(BaseConfig oldBaseConfig) {
if (!Strings.isNullOrEmpty(urlBase) && (!urlBase.startsWith("/") || urlBase.endsWith("/") || "/".equals(urlBase))) {
if (!urlBase.startsWith("/")) {
urlBase = "/" + urlBase;
}
if (urlBase.endsWith("/")) {
urlBase = urlBase.substring(0, urlBase.length() - 1);
}
if ("/".equals(urlBase) || "".equals(urlBase)) {
urlBase = "/";
}
setUrlBase(urlBase);
}
return this;
}
@Override
public MainConfig updateAfterLoading() {
return this;
}
@Override
public MainConfig initializeNewConfig() {
Random random = new Random();
setApiKey(new BigInteger(130, random).toString(32).toUpperCase());
return this;
}
}

70
core/src/main/java/org/nzbhydra/config/NotificationConfig.java
View File

@ -1,70 +0,0 @@
package org.nzbhydra.config;
import joptsimple.internal.Strings;
import lombok.Data;
import org.nzbhydra.config.sensitive.SensitiveData;
import org.springframework.boot.context.properties.ConfigurationProperties;
import java.util.ArrayList;
import java.util.List;
@SuppressWarnings("unchecked")
@ConfigurationProperties
@Data
public class NotificationConfig extends ValidatingConfig<NotificationConfig> {
public enum AppriseType {
NONE,
API,
CLI
}
private AppriseType appriseType = AppriseType.NONE;
@SensitiveData
private String appriseApiUrl;
@SensitiveData
private String appriseCliPath;
private boolean displayNotifications;
private int displayNotificationsMax;
private List<NotificationConfigEntry> entries = new ArrayList<>();
private List<String> filterOuts = new ArrayList<>();
public NotificationConfig() {
}
@Override
public ConfigValidationResult validateConfig(BaseConfig oldConfig, NotificationConfig newConfig, BaseConfig newBaseConfig) {
final List<String> errors = new ArrayList<>();
final List<String> warnings = new ArrayList<>();
if (newBaseConfig.getNotificationConfig().getEntries().stream()
.anyMatch(x -> Strings.isNullOrEmpty(x.getAppriseUrls()))) {
errors.add("Make sure all notification entries contain a URL");
}
final boolean appriseUrlSet = !Strings.isNullOrEmpty(newBaseConfig.getNotificationConfig().getAppriseApiUrl());
final boolean anyEntries = newBaseConfig.getNotificationConfig().getEntries().isEmpty();
if (anyEntries && !appriseUrlSet) {
warnings.add("No notifications will be sent unless the Apprise API URL is configured.");
}
return new ConfigValidationResult(true, false, errors, warnings);
}
@Override
public NotificationConfig prepareForSaving(BaseConfig oldBaseConfig) {
return this;
}
@Override
public NotificationConfig updateAfterLoading() {
return this;
}
@Override
public NotificationConfig initializeNewConfig() {
return this;
}
}

33
core/src/main/java/org/nzbhydra/config/NotificationConfigEntry.java
View File

@ -1,33 +0,0 @@
package org.nzbhydra.config;
import com.fasterxml.jackson.annotation.JsonFormat;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import org.nzbhydra.config.sensitive.SensitiveData;
import org.nzbhydra.notifications.NotificationEventType;
@SuppressWarnings("unchecked")
@Data
@NoArgsConstructor
@AllArgsConstructor
public class NotificationConfigEntry {
public enum MessageType {
INFO,
SUCCESS,
WARNING,
FAILURE
}
@JsonFormat(shape = JsonFormat.Shape.STRING)
private NotificationEventType eventType;
@SensitiveData
private String appriseUrls;
private String titleTemplate;
private String bodyTemplate;
@JsonFormat(shape = JsonFormat.Shape.STRING)
private MessageType messageType;
}

30
core/src/main/java/org/nzbhydra/config/SearchSourceRestriction.java
View File

@ -1,30 +0,0 @@
package org.nzbhydra.config;
import org.nzbhydra.searching.searchrequests.SearchRequest;
public enum SearchSourceRestriction {
INTERNAL,
API,
ALL_BUT_RSS,
ONLY_RSS,
BOTH,
NONE;
public boolean meets(SearchRequest searchRequest) {
if (this == ALL_BUT_RSS && searchRequest.getSource() == SearchRequest.SearchSource.API) {
return searchRequest.getQuery().isPresent() || !searchRequest.getIdentifiers().isEmpty();
}
if (this == ONLY_RSS && searchRequest.getSource() == SearchRequest.SearchSource.API) {
return !searchRequest.getQuery().isPresent() && searchRequest.getIdentifiers().isEmpty();
}
return meets(searchRequest.getSource());
}
public boolean meets(SearchRequest.SearchSource searchSource) {
return searchSource.name().equals(this.name()) || this == BOTH || this == ALL_BUT_RSS || (this == ONLY_RSS && searchSource == SearchRequest.SearchSource.API);
}
public boolean isEnabled() {
return this != NONE;
}
}

168
core/src/main/java/org/nzbhydra/config/SearchingConfig.java
View File

@ -1,168 +0,0 @@
package org.nzbhydra.config;
import com.fasterxml.jackson.annotation.JsonFormat;
import com.fasterxml.jackson.annotation.JsonFormat.Shape;
import com.fasterxml.jackson.annotation.JsonSetter;
import com.google.common.base.Strings;
import lombok.Data;
import org.nzbhydra.indexers.QueryGenerator;
import org.nzbhydra.searching.CustomQueryAndTitleMapping;
import org.nzbhydra.searching.searchrequests.SearchRequest;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.context.properties.ConfigurationProperties;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import java.util.Optional;
import java.util.Set;
import java.util.stream.Collectors;
@SuppressWarnings("unchecked")
@Data
@ConfigurationProperties
public class SearchingConfig extends ValidatingConfig<SearchingConfig> {
private static final Logger logger = LoggerFactory.getLogger(SearchingConfig.class);
@JsonFormat(shape = Shape.STRING)
private SearchSourceRestriction applyRestrictions = SearchSourceRestriction.BOTH;
private int coverSize = 128;
private List<CustomQueryAndTitleMapping.Mapping> customMappings = new ArrayList<>();
private Integer globalCacheTimeMinutes;
private float duplicateAgeThreshold = 2.0F;
private float duplicateSizeThresholdInPercent = 1.0F;
private List<String> forbiddenGroups = new ArrayList<>();
private List<String> forbiddenPosters = new ArrayList<>();
private String forbiddenRegex;
private List<String> forbiddenWords = new ArrayList<>();
private SearchSourceRestriction alwaysConvertIds = SearchSourceRestriction.NONE;
private SearchSourceRestriction generateQueries = SearchSourceRestriction.INTERNAL;
private QueryGenerator.QueryFormat generateQueriesFormat = QueryGenerator.QueryFormat.TITLE;
@JsonFormat(shape = Shape.STRING)
private SearchSourceRestriction idFallbackToQueryGeneration = SearchSourceRestriction.NONE;
private boolean ignorePassworded = false;
private boolean ignoreTemporarilyDisabled = false;
private boolean ignoreLoadLimitingForInternalSearches = false;
private int keepSearchResultsForDays = 3;
private String language = "en";
private List<String> languagesToKeep = new ArrayList<>();
private boolean loadAllCachedOnInternal;
private int loadLimitInternal = 100;
private Integer maxAge;
private Integer minSeeders;
@JsonSetter()
private List<String> removeTrailing = new ArrayList<>();
private boolean replaceUmlauts = false;
private String requiredRegex;
private List<String> requiredWords = new ArrayList<>();
private boolean sendTorznabCategories = true;
private boolean showQuickFilterButtons = true;
private boolean alwaysShowQuickFilterButtons = false;
private List<String> customQuickFilterButtons = new ArrayList<>();
private List<String> preselectQuickFilterButtons = new ArrayList<>();
private Integer timeout = 30;
private boolean transformNewznabCategories = true;
private String userAgent = "NZBHydra2";
private List<String> userAgents = new ArrayList<>(Arrays.asList("Mozilla", "Sonarr", "Radarr", "CouchPotato", "LazyLibrarian", "Lidarr", "NZBGet", "sabNZBd", "Readarr"));
private boolean useOriginalCategories = false;
private boolean wrapApiErrors = false;
public SearchingConfig() {
}
public Optional<Integer> getGlobalCacheTimeMinutes() {
return Optional.ofNullable(globalCacheTimeMinutes);
}
public Optional<Integer> getMaxAge() {
return Optional.ofNullable(maxAge);
}
public Optional<String> getForbiddenRegex() {
return Optional.ofNullable(Strings.emptyToNull(forbiddenRegex));
}
public Optional<String> getRequiredRegex() {
return Optional.ofNullable(Strings.emptyToNull(requiredRegex));
}
public Optional<String> getUserAgent() {
return Optional.ofNullable(Strings.emptyToNull(userAgent));
}
public Optional<String> getLanguage() {
return Optional.ofNullable(Strings.emptyToNull(language));
}
@Override
public ConfigValidationResult validateConfig(BaseConfig oldConfig, SearchingConfig newConfig, BaseConfig newBaseConfig) {
List<String> errors = new ArrayList<>();
List<String> warnings = new ArrayList<>();
checkRegex(errors, requiredRegex, "The required regex in \"Searching\" is invalid");
checkRegex(errors, forbiddenRegex, "The forbidden in \"Searching\" is invalid");
if (applyRestrictions == SearchSourceRestriction.NONE) {
if (!getRequiredWords().isEmpty() || !getForbiddenWords().isEmpty()) {
warnings.add("You selected not to apply any word restrictions in \"Searching\" but supplied forbidden or required words there");
}
if (getRequiredRegex().isPresent() || getForbiddenRegex().isPresent()) {
warnings.add("You selected not to apply any word restrictions in \"Searching\" but supplied a forbidden or required regex there");
}
}
final CustomQueryAndTitleMapping customQueryAndTitleMapping = new CustomQueryAndTitleMapping(newBaseConfig);
final SearchRequest searchRequest = new SearchRequest();
searchRequest.setTitle("test title");
searchRequest.setQuery("test query");
for (CustomQueryAndTitleMapping.Mapping customMapping : newConfig.getCustomMappings()) {
try {
customQueryAndTitleMapping.mapSearchRequest(searchRequest, Collections.singletonList(customMapping));
} catch (Exception e) {
errors.add(String.format("Unable to process mapping %s:}\n%s", customMapping.toString(), e.getMessage()));
}
if (customMapping.getFrom().contains("{episode:")) {
errors.add("The group 'episode' is not allowed in custom mapping input patterns.");
}
if (customMapping.getFrom().contains("{season:")) {
errors.add("The group 'season' is not allowed in custom mapping input patterns.");
}
}
final List<String> emptyTrailing = (newConfig.getRemoveTrailing().stream().filter(Strings::isNullOrEmpty)).collect(Collectors.toList());
if (!emptyTrailing.isEmpty()) {
errors.add("Trailing values to remove contains empty values");
}
return new ConfigValidationResult(errors.isEmpty(), false, errors, warnings);
}
@Override
public SearchingConfig prepareForSaving(BaseConfig oldBaseConfig) {
final Set<String> customQuickfilterNames = customQuickFilterButtons.stream().map(x -> x.split("=")[0]).collect(Collectors.toSet());
for (Iterator<String> iterator = getPreselectQuickFilterButtons().iterator(); iterator.hasNext(); ) {
String preselectQuickFilterButton = iterator.next();
final String[] split = preselectQuickFilterButton.split("\\|");
if ("custom".equals(split[0]) && !customQuickfilterNames.contains(split[0])) {
logger.info("Custom quickfilter {} doesn't exist anymore, removing it from list of filters to preselect.", preselectQuickFilterButton);
iterator.remove();
}
}
return this;
}
@Override
public SearchingConfig updateAfterLoading() {
return this;
}
@Override
public SearchingConfig initializeNewConfig() {
return this;
}
}

108
core/src/main/java/org/nzbhydra/config/ValidatingConfig.java
View File

@ -1,108 +0,0 @@
package org.nzbhydra.config;
import com.google.common.base.Strings;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.lang.reflect.Field;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Objects;
import java.util.regex.Pattern;
import java.util.regex.PatternSyntaxException;
public abstract class ValidatingConfig<T> {
private static final Logger logger = LoggerFactory.getLogger(ValidatingConfig.class);
/**
* @param oldConfig old config state (e.g. to compare what has changed)
* @param newConfig the new config. Will always be the same object as the one on which the method was called
* @param newBaseConfig
* @return a list of error messages or an empty list when everything is fine
*/
public abstract ConfigValidationResult validateConfig(BaseConfig oldConfig, T newConfig, BaseConfig newBaseConfig);
@Data
@AllArgsConstructor
@NoArgsConstructor
public static class ConfigValidationResult {
private boolean ok = true;
private boolean restartNeeded;
private List<String> errorMessages = new ArrayList<>();
private List<String> warningMessages = new ArrayList<>();
private BaseConfig newConfig;
public ConfigValidationResult(boolean ok, boolean restartNeeded, List<String> errorMessages, List<String> warningMessages) {
this.ok = ok;
this.restartNeeded = restartNeeded;
this.errorMessages.addAll(new HashSet<>(errorMessages));
this.warningMessages.addAll(new HashSet<>(warningMessages));
}
}
protected void checkRegex(List<String> errorMessages, String regex, String errorMessage) {
if (!Strings.isNullOrEmpty(regex)) {
try {
Pattern.compile(regex);
} catch (PatternSyntaxException e) {
errorMessages.add(errorMessage);
}
}
}
/**
* Detects if any setting was changed that requires a restart to be effective
*
* @param configToCompare the old config (its settings will be compared with the ones from the calling instance)
* @return
*/
protected boolean isRestartNeeded(Object configToCompare) {
for (Field field : configToCompare.getClass().getDeclaredFields()) {
if (field.isAnnotationPresent(RestartRequired.class)) {
try {
//PropertyDescriptor doesn't work for some reason, this is just as fine for what we need
String getterName = (field.getType() == Boolean.class || field.getType() == boolean.class ? "is" : "get") + field.getName().substring(0, 1).toUpperCase() + field.getName().substring(1);
Method method = configToCompare.getClass().getDeclaredMethod(getterName);
Object oldValue = method.invoke(configToCompare);
Object newValue = method.invoke(this);
if (!Objects.equals(oldValue, newValue)) {
logger.debug("Restart needed because field {} has changed", field.getName());
return true;
}
} catch (InvocationTargetException | NoSuchMethodException | IllegalAccessException e) {
logger.error("Unable to determine if field '{}' in class {} was changed", field.getName(), configToCompare.getClass().getName());
}
}
}
return false;
}
/**
* Called before the config is saved after the user made some changes. Use this to convert data, e.g. passwords.
*
* @param oldBaseConfig
*/
public abstract T prepareForSaving(BaseConfig oldBaseConfig);
/**
* Called before the config is transferred to the GUI. Use this to prepare data, e.g. passwords.
*/
public abstract T updateAfterLoading();
/**
* Called for a new config to initialize itself
*/
public abstract T initializeNewConfig();
}

118
core/src/main/java/org/nzbhydra/config/auth/AuthConfig.java
View File

@ -1,118 +0,0 @@
/*
* (C) Copyright 2017 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra.config.auth;
import com.fasterxml.jackson.annotation.JsonFormat;
import com.fasterxml.jackson.annotation.JsonFormat.Shape;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.google.common.base.Joiner;
import lombok.Data;
import lombok.EqualsAndHashCode;
import org.javers.core.metamodel.annotation.DiffIgnore;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.config.RestartRequired;
import org.nzbhydra.config.ValidatingConfig;
import org.nzbhydra.config.sensitive.SensitiveData;
import org.springframework.boot.context.properties.ConfigurationProperties;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
@Data
@ConfigurationProperties
@EqualsAndHashCode
public class AuthConfig extends ValidatingConfig<AuthConfig> {
@JsonFormat(shape = Shape.STRING)
@RestartRequired
private AuthType authType;
private boolean rememberUsers = true;
private int rememberMeValidityDays;
@SensitiveData
private String authHeader;
private List<String> authHeaderIpRanges = new ArrayList<>();
private boolean restrictAdmin = false;
private boolean restrictDetailsDl = false;
private boolean restrictIndexerSelection = false;
private boolean restrictSearch = false;
private boolean restrictStats = false;
private boolean allowApiStats = true;
@DiffIgnore
private List<UserAuthConfig> users = new ArrayList<>();
@JsonIgnore
public boolean isAuthConfigured() {
return authType != AuthType.NONE;
}
@Override
public ConfigValidationResult validateConfig(BaseConfig oldConfig, AuthConfig newConfig, BaseConfig newBaseConfig) {
List<String> errors = new ArrayList<>();
List<String> warnings = new ArrayList<>();
if (authType != AuthType.NONE && users.isEmpty()) {
errors.add("You've enabled security but not defined any users");
} else if (authType != AuthType.NONE && restrictAdmin && users.stream().noneMatch(UserAuthConfig::isMaySeeAdmin)) {
errors.add("You've restricted admin access but no user has admin rights");
} else if (authType != AuthType.NONE && !restrictSearch && !restrictAdmin) {
errors.add("You haven't enabled any access restrictions. Auth will not take any effect");
}
Set<String> usernames = new HashSet<>();
List<String> duplicateUsernames = new ArrayList<>();
for (UserAuthConfig user : users) {
if (usernames.contains(user.getUsername())) {
duplicateUsernames.add(user.getUsername());
}
usernames.add(user.getUsername());
}
if (!duplicateUsernames.isEmpty()) {
errors.add("The following user names are not unique: " + Joiner.on(", ").join(duplicateUsernames));
}
if (!authHeaderIpRanges.isEmpty()) {
authHeaderIpRanges.forEach(x -> {
Matcher matcher = Pattern.compile("^((?:[0-9]{1,3}\\.){3}[0-9]{1,3}(-(?:[0-9]{1,3}\\.){3}[0-9]{1,3})?,?)+$").matcher(x);
if (!matcher.matches()) {
errors.add("IP range " + x + " is invalid");
}
});
}
return new ConfigValidationResult(errors.isEmpty(), isRestartNeeded(oldConfig.getAuth()), errors, warnings);
}
@Override
public AuthConfig prepareForSaving(BaseConfig oldBaseConfig) {
getUsers().forEach(userAuthConfig -> userAuthConfig.prepareForSaving(oldBaseConfig));
return this;
}
@Override
public AuthConfig updateAfterLoading() {
getUsers().forEach(ValidatingConfig::updateAfterLoading);
return this;
}
@Override
public AuthConfig initializeNewConfig() {
return this;
}
}

1
core/src/main/java/org/nzbhydra/config/migration/ConfigMigrationStep006to007.java
View File

@ -22,7 +22,6 @@ import org.slf4j.LoggerFactory;
import java.util.List;
import java.util.Map;
@SuppressWarnings("unchecked")
public class ConfigMigrationStep006to007 implements ConfigMigrationStep {
private static final Logger logger = LoggerFactory.getLogger(ConfigMigrationStep006to007.class);

1
core/src/main/java/org/nzbhydra/config/migration/ConfigMigrationStep007to008.java
View File

@ -21,7 +21,6 @@ import org.slf4j.LoggerFactory;
import java.util.Map;
@SuppressWarnings("unchecked")
public class ConfigMigrationStep007to008 implements ConfigMigrationStep {
private static final Logger logger = LoggerFactory.getLogger(ConfigMigrationStep007to008.class);

1
core/src/main/java/org/nzbhydra/config/migration/ConfigMigrationStep008to009.java
View File

@ -21,7 +21,6 @@ import org.slf4j.LoggerFactory;
import java.util.Map;
@SuppressWarnings("unchecked")
public class ConfigMigrationStep008to009 implements ConfigMigrationStep {
private static final Logger logger = LoggerFactory.getLogger(ConfigMigrationStep008to009.class);

1
core/src/main/java/org/nzbhydra/config/migration/ConfigMigrationStep009to010.java
View File

@ -19,7 +19,6 @@ package org.nzbhydra.config.migration;
import java.util.ArrayList;
import java.util.Map;
@SuppressWarnings("unchecked")
public class ConfigMigrationStep009to010 implements ConfigMigrationStep {
@Override

1
core/src/main/java/org/nzbhydra/config/migration/ConfigMigrationStep010to011.java
View File

@ -18,7 +18,6 @@ package org.nzbhydra.config.migration;
import java.util.Map;
@SuppressWarnings("unchecked")
public class ConfigMigrationStep010to011 implements ConfigMigrationStep {
@Override

1
core/src/main/java/org/nzbhydra/config/migration/ConfigMigrationStep011to012.java
View File

@ -21,7 +21,6 @@ import org.slf4j.LoggerFactory;
import java.util.Map;
@SuppressWarnings("unchecked")
public class ConfigMigrationStep011to012 implements ConfigMigrationStep {
private static final Logger logger = LoggerFactory.getLogger(ConfigMigration.class);

1
core/src/main/java/org/nzbhydra/config/migration/ConfigMigrationStep012to013.java
View File

@ -21,7 +21,6 @@ import org.slf4j.LoggerFactory;
import java.util.Map;
@SuppressWarnings("unchecked")
public class ConfigMigrationStep012to013 implements ConfigMigrationStep {
private static final Logger logger = LoggerFactory.getLogger(ConfigMigration.class);

1
core/src/main/java/org/nzbhydra/config/migration/ConfigMigrationStep013to014.java
View File

@ -21,7 +21,6 @@ import org.slf4j.LoggerFactory;
import java.util.Map;
@SuppressWarnings("unchecked")
public class ConfigMigrationStep013to014 implements ConfigMigrationStep {
private static final Logger logger = LoggerFactory.getLogger(ConfigMigration.class);

1
core/src/main/java/org/nzbhydra/config/migration/ConfigMigrationStep018to019.java
View File

@ -21,7 +21,6 @@ import org.slf4j.LoggerFactory;
import java.util.Map;
@SuppressWarnings("unchecked")
public class ConfigMigrationStep018to019 implements ConfigMigrationStep {
private static final Logger logger = LoggerFactory.getLogger(ConfigMigration.class);

2
core/src/main/java/org/nzbhydra/config/safeconfig/SafeCategory.java
View File

@ -2,9 +2,11 @@ package org.nzbhydra.config.safeconfig;
import lombok.Data;
import org.nzbhydra.config.category.Category;
import org.nzbhydra.springnative.ReflectionMarker;
//Only needed because I can't convince Thymeleaf to serialize enums as their names
@Data
@ReflectionMarker
public class SafeCategory {
private final boolean mayBeSelected;

2
core/src/main/java/org/nzbhydra/config/safeconfig/SafeConfig.java
View File

@ -3,11 +3,13 @@ package org.nzbhydra.config.safeconfig;
import lombok.Data;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.config.auth.AuthType;
import org.nzbhydra.springnative.ReflectionMarker;
import java.util.List;
import java.util.stream.Collectors;
@Data
@ReflectionMarker
public class SafeConfig {
private SafeCategoriesConfig categoriesConfig;

2
core/src/main/java/org/nzbhydra/config/safeconfig/SafeDownloaderConfig.java
View File

@ -2,8 +2,10 @@ package org.nzbhydra.config.safeconfig;
import lombok.Data;
import org.nzbhydra.config.downloading.DownloaderConfig;
import org.nzbhydra.springnative.ReflectionMarker;
@Data
@ReflectionMarker
public class SafeDownloaderConfig {
private String defaultCategory;

2
core/src/main/java/org/nzbhydra/config/safeconfig/SafeIndexerConfig.java
View File

@ -3,10 +3,12 @@ package org.nzbhydra.config.safeconfig;
import lombok.Data;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.config.indexer.IndexerConfig;
import org.nzbhydra.springnative.ReflectionMarker;
import java.util.List;
@Data
@ReflectionMarker
public class SafeIndexerConfig {
private String name;

3
core/src/main/java/org/nzbhydra/config/sensitive/SensitiveDataHidingSerializer.java
View File

@ -16,8 +16,7 @@ public class SensitiveDataHidingSerializer extends JsonSerializer<Object> {
@Override
public void serialize(Object value, JsonGenerator gen, SerializerProvider serializers) throws IOException {
String toWrite = "<REMOVED>";
if (value instanceof Optional) {
Optional optional = (Optional) value;
if (value instanceof Optional optional) {
toWrite = optional.isPresent() ? "<REMOVED>" : "<NOTSET>";
}
logger.debug("Hiding sensitive data in config setting \"{}\"", gen.getOutputContext().getCurrentName());

92
core/src/main/java/org/nzbhydra/config/validation/AuthConfigValidator.java Normal file
View File

@ -0,0 +1,92 @@
/*
* (C) Copyright 2023 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra.config.validation;
import com.google.common.base.Joiner;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.config.auth.AuthConfig;
import org.nzbhydra.config.auth.AuthType;
import org.nzbhydra.config.auth.UserAuthConfig;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
@Component
public class AuthConfigValidator implements ConfigValidator<AuthConfig> {
@Autowired
private UserAuthConfigValidator userAuthConfigValidator;
@Override
public boolean doesValidate(Class<?> clazz) {
return clazz == AuthConfig.class;
}
@Override
public ConfigValidationResult validateConfig(BaseConfig oldBaseConfig, BaseConfig newBaseConfig, AuthConfig newConfig) {
List<String> errors = new ArrayList<>();
List<String> warnings = new ArrayList<>();
if (newConfig.getAuthType() != AuthType.NONE && newConfig.getUsers().isEmpty()) {
errors.add("You've enabled security but not defined any users");
} else if (newConfig.getAuthType() != AuthType.NONE && newConfig.isRestrictAdmin() && newConfig.getUsers().stream().noneMatch(UserAuthConfig::isMaySeeAdmin)) {
errors.add("You've restricted admin access but no user has admin rights");
} else if (newConfig.getAuthType() != AuthType.NONE && !newConfig.isRestrictSearch() && !newConfig.isRestrictAdmin()) {
errors.add("You haven't enabled any access restrictions. Auth will not take any effect");
}
Set<String> usernames = new HashSet<>();
List<String> duplicateUsernames = new ArrayList<>();
for (UserAuthConfig user : newConfig.getUsers()) {
if (usernames.contains(user.getUsername())) {
duplicateUsernames.add(user.getUsername());
}
usernames.add(user.getUsername());
}
if (!duplicateUsernames.isEmpty()) {
errors.add("The following user names are not unique: " + Joiner.on(", ").join(duplicateUsernames));
}
if (!newConfig.getAuthHeaderIpRanges().isEmpty()) {
newConfig.getAuthHeaderIpRanges().forEach(x -> {
Matcher matcher = Pattern.compile("^((?:[0-9]{1,3}\\.){3}[0-9]{1,3}(-(?:[0-9]{1,3}\\.){3}[0-9]{1,3})?,?)+$").matcher(x);
if (!matcher.matches()) {
errors.add("IP range " + x + " is invalid");
}
});
}
return new ConfigValidationResult(errors.isEmpty(), ConfigValidationTools.isRestartNeeded(oldBaseConfig.getAuth(), newConfig), errors, warnings);
}
@Override
public AuthConfig prepareForSaving(BaseConfig oldBaseConfig, AuthConfig newAuthConfig) {
newAuthConfig.getUsers().forEach(userAuthConfig -> userAuthConfigValidator.prepareForSaving(oldBaseConfig, userAuthConfig));
return new AuthConfig();
}
@Override
public AuthConfig updateAfterLoading(AuthConfig newAuthConfig) {
newAuthConfig.getUsers().forEach(userAuthConfig -> userAuthConfigValidator.updateAfterLoading(userAuthConfig));
return newAuthConfig;
}
}

172
core/src/main/java/org/nzbhydra/config/validation/BaseConfigValidator.java Normal file
View File

@ -0,0 +1,172 @@
/*
* (C) Copyright 2023 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra.config.validation;
import com.google.common.base.Joiner;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.config.SearchSourceRestriction;
import org.nzbhydra.config.indexer.IndexerConfig;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;
@SuppressWarnings("unchecked")
@Component
public class BaseConfigValidator implements ConfigValidator<BaseConfig> {
private static final Logger logger = LoggerFactory.getLogger(BaseConfigValidator.class);
@Autowired
private CategoriesConfigValidator categoriesConfigValidator;
@Autowired
private DownloadingConfigValidator downloadingConfigValidator;
@Autowired
private SearchingConfigValidator searchingConfigValidator;
@Autowired
private MainConfigValidator mainConfigValidator;
@Autowired
private AuthConfigValidator authConfigValidator;
@Autowired
private IndexerConfigValidator indexerConfigValidator;
@Autowired
private List<ConfigValidator> configValidatorList;
@Override
public boolean doesValidate(Class<?> clazz) {
return clazz == BaseConfig.class;
}
@Override
public ConfigValidationResult validateConfig(BaseConfig oldBaseConfig, BaseConfig newBaseConfig, BaseConfig newConfig) {
final ConfigValidationResult configValidationResult = new ConfigValidationResult();
final List<Object> configs = new ArrayList<>(Arrays.asList(
newBaseConfig.getMain(),
newBaseConfig.getSearching(),
newBaseConfig.getDownloading(),
newBaseConfig.getCategoriesConfig(),
newBaseConfig.getAuth()
));
configs.addAll(newConfig.getIndexers());
for (Object config : configs) {
final ConfigValidator validator = configValidatorList.stream().filter(x -> x.doesValidate(config.getClass())).findFirst().orElseThrow();
final ConfigValidationResult result = validator.validateConfig(oldBaseConfig, newBaseConfig, config);
configValidationResult.getErrorMessages().addAll(result.getErrorMessages());
configValidationResult.getWarningMessages().addAll(result.getWarningMessages());
}
validateIndexers(newConfig, configValidationResult);
if (!configValidationResult.getErrorMessages().isEmpty()) {
logger.warn("Config validation returned errors:\n" + Joiner.on("\n").join(configValidationResult.getErrorMessages()));
}
if (!configValidationResult.getWarningMessages().isEmpty()) {
logger.warn("Config validation returned warnings:\n" + Joiner.on("\n").join(configValidationResult.getWarningMessages()));
}
if (configValidationResult.isRestartNeeded()) {
logger.warn("Settings were changed that require a restart to become effective");
}
configValidationResult.setOk(configValidationResult.getErrorMessages().isEmpty());
return configValidationResult;
}
private void validateIndexers(BaseConfig newConfig, ConfigValidationResult configValidationResult) {
if (!newConfig.getIndexers().isEmpty()) {
if (newConfig.getIndexers().stream().noneMatch(x -> x.getState() == IndexerConfig.State.ENABLED)) {
configValidationResult.getWarningMessages().add("No indexers enabled. Searches will return empty results");
} else if (newConfig.getIndexers().stream().allMatch(x -> x.getSupportedSearchIds().isEmpty())) {
if (newConfig.getSearching().getGenerateQueries() == SearchSourceRestriction.NONE) {
configValidationResult.getWarningMessages().add("No indexer found that supports search IDs. Without query generation searches using search IDs will return empty results.");
} else if (newConfig.getSearching().getGenerateQueries() != SearchSourceRestriction.BOTH) {
String name = newConfig.getSearching().getGenerateQueries() == SearchSourceRestriction.API ? "internal" : "API";
configValidationResult.getWarningMessages().add("No indexer found that supports search IDs. Without query generation " + name + " searches using search IDs will return empty results.");
}
}
Set<String> indexerNames = new HashSet<>();
Set<String> duplicateIndexerNames = new HashSet<>();
for (IndexerConfig indexer : newConfig.getIndexers()) {
if (!indexerNames.add(indexer.getName())) {
duplicateIndexerNames.add(indexer.getName());
}
}
if (!duplicateIndexerNames.isEmpty()) {
configValidationResult.getErrorMessages().add("Duplicate indexer names found: " + Joiner.on(", ").join(duplicateIndexerNames));
}
final Set<Set<String>> indexersSameHostAndApikey = new HashSet<>();
for (IndexerConfig indexer : newConfig.getIndexers()) {
final Set<String> otherIndexersSameHostAndApiKey = newConfig.getIndexers().stream()
.filter(x -> x != indexer)
.filter(x -> IndexerConfig.isIndexerEquals(x, indexer))
.map(IndexerConfig::getName)
.collect(Collectors.toSet());
if (!otherIndexersSameHostAndApiKey.isEmpty()) {
otherIndexersSameHostAndApiKey.add(indexer.getName());
if (indexersSameHostAndApikey.stream().noneMatch(x -> x.contains(indexer.getName()))) {
indexersSameHostAndApikey.add(otherIndexersSameHostAndApiKey);
final String message = "Found multiple indexers with same host and API key: " + Joiner.on(", ").join(otherIndexersSameHostAndApiKey);
logger.warn(message);
configValidationResult.getWarningMessages().add(message);
}
}
}
} else {
configValidationResult.getWarningMessages().add("No indexers configured. You won't get any results");
}
}
@Override
public BaseConfig prepareForSaving(BaseConfig oldBaseConfig, BaseConfig newConfig) {
categoriesConfigValidator.prepareForSaving(oldBaseConfig, newConfig.getCategoriesConfig());
downloadingConfigValidator.prepareForSaving(oldBaseConfig, newConfig.getDownloading());
searchingConfigValidator.prepareForSaving(oldBaseConfig, newConfig.getSearching());
mainConfigValidator.prepareForSaving(oldBaseConfig, newConfig.getMain());
authConfigValidator.prepareForSaving(oldBaseConfig, newConfig.getAuth());
newConfig.getIndexers().forEach(x -> indexerConfigValidator.prepareForSaving(oldBaseConfig, x));
return newConfig;
}
@Override
public BaseConfig updateAfterLoading(BaseConfig newConfig) {
authConfigValidator.updateAfterLoading(newConfig.getAuth());
return newConfig;
}
@Override
public BaseConfig initializeNewConfig(BaseConfig newConfig) {
categoriesConfigValidator.initializeNewConfig(newConfig.getCategoriesConfig());
downloadingConfigValidator.initializeNewConfig(newConfig.getDownloading());
searchingConfigValidator.initializeNewConfig(newConfig.getSearching());
mainConfigValidator.initializeNewConfig(newConfig.getMain());
authConfigValidator.initializeNewConfig(newConfig.getAuth());
return newConfig;
}
}

79
core/src/main/java/org/nzbhydra/config/category/CategoriesConfig.java → core/src/main/java/org/nzbhydra/config/validation/CategoriesConfigValidator.java
View File

@ -1,5 +1,5 @@
/*
* (C) Copyright 2017 TheOtherP (theotherp@posteo.net)
* (C) Copyright 2023 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@ -14,50 +14,38 @@
* limitations under the License.
*/
package org.nzbhydra.config.category;
package org.nzbhydra.config.validation;
import com.google.common.base.Joiner;
import com.google.common.base.MoreObjects;
import lombok.Data;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.config.SearchSourceRestriction;
import org.nzbhydra.config.ValidatingConfig;
import org.nzbhydra.config.category.Category.Subtype;
import org.nzbhydra.config.category.CategoriesConfig;
import org.nzbhydra.config.category.Category;
import org.nzbhydra.searching.CategoryProvider;
import org.springframework.stereotype.Component;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
import java.util.Optional;
import java.util.stream.Collectors;
import static org.nzbhydra.searching.dtoseventsenums.SearchType.SEARCH;
import static org.nzbhydra.config.validation.ConfigValidationTools.checkRegex;
@Data
public class CategoriesConfig extends ValidatingConfig<CategoriesConfig> {
public final static Category allCategory = new Category("All");
static {
allCategory.setApplyRestrictionsType(SearchSourceRestriction.NONE);
allCategory.setIgnoreResultsFrom(SearchSourceRestriction.NONE);
allCategory.setMayBeSelected(true);
allCategory.setSearchType(SEARCH);
allCategory.setSubtype(Subtype.ALL);
}
private boolean enableCategorySizes = true;
private List<Category> categories = new ArrayList<>();
private String defaultCategory = "All";
@Component
public class CategoriesConfigValidator implements ConfigValidator<CategoriesConfig> {
@Override
public ConfigValidationResult validateConfig(BaseConfig oldConfig, CategoriesConfig newConfig, BaseConfig newBaseConfig) {
public boolean doesValidate(Class<?> clazz) {
return clazz == CategoriesConfig.class;
}
@Override
public ConfigValidationResult validateConfig(BaseConfig oldBaseConfig, BaseConfig newBaseConfig, CategoriesConfig newConfig) {
ArrayList<String> errors = new ArrayList<>();
ArrayList<String> warnings = new ArrayList<>();
for (Category category : categories) {
for (Category category : newConfig.getCategories()) {
if (category.getNewznabCategories() == null || category.getNewznabCategories().isEmpty()) {
errors.add("Category \"" + category.getName() + "\" does not have any newznab categories configured");
} else {
@ -86,49 +74,16 @@ public class CategoriesConfig extends ValidatingConfig<CategoriesConfig> {
}
}
}
List<Integer> allNewznabCategories = categories.stream().flatMap(x -> x.getNewznabCategories().stream().flatMap(Collection::stream)).collect(Collectors.toList());
List<Integer> allNewznabCategories = newConfig.getCategories().stream().flatMap(x -> x.getNewznabCategories().stream().flatMap(Collection::stream)).toList();
List<Integer> duplicateNewznabCategories = allNewznabCategories.stream().filter(x -> Collections.frequency(allNewznabCategories, 1) > 1).collect(Collectors.toList());
if (!duplicateNewznabCategories.isEmpty()) {
errors.add("The following newznab categories are assigned to multiple indexers: " + Joiner.on(", ").join(duplicateNewznabCategories));
}
if (!"All".equals(newConfig.getDefaultCategory()) && categories.stream().noneMatch(x -> x.getName().equals(newConfig.getDefaultCategory()))) {
if (!"All".equals(newConfig.getDefaultCategory()) && newConfig.getCategories().stream().noneMatch(x -> x.getName().equals(newConfig.getDefaultCategory()))) {
errors.add("Category \"" + newConfig.getDefaultCategory() + "\" set as default category but no such category exists");
}
return new ConfigValidationResult(errors.isEmpty(), false, errors, warnings);
}
public void setCategories(List<Category> categories) {
categories.sort(Comparator.comparing(Category::getName));
this.categories = categories;
}
public List<Category> withoutAll() {
return categories.stream().filter(x -> !allCategory.equals(x)).collect(Collectors.toList());
}
@Override
public CategoriesConfig prepareForSaving(BaseConfig oldBaseConfig) {
return this;
}
@Override
public CategoriesConfig updateAfterLoading() {
return this;
}
@Override
public CategoriesConfig initializeNewConfig() {
return this;
}
@Override
public String toString() {
return MoreObjects.toStringHelper(this)
.add("enableCategorySizes", enableCategorySizes)
.add("categories", categories)
.add("defaultCategory", defaultCategory)
.toString();
}
}

46
core/src/main/java/org/nzbhydra/config/validation/ConfigValidationResult.java Normal file
View File

@ -0,0 +1,46 @@
/*
* (C) Copyright 2023 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra.config.validation;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.springnative.ReflectionMarker;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
@Data
@ReflectionMarker
@AllArgsConstructor
@NoArgsConstructor
public class ConfigValidationResult {
private boolean ok = true;
private boolean restartNeeded;
private List<String> errorMessages = new ArrayList<>();
private List<String> warningMessages = new ArrayList<>();
private BaseConfig newConfig;
public ConfigValidationResult(boolean ok, boolean restartNeeded, List<String> errorMessages, List<String> warningMessages) {
this.ok = ok;
this.restartNeeded = restartNeeded;
this.errorMessages.addAll(new HashSet<>(errorMessages));
this.warningMessages.addAll(new HashSet<>(warningMessages));
}
}

73
core/src/main/java/org/nzbhydra/config/validation/ConfigValidationTools.java Normal file
View File

@ -0,0 +1,73 @@
/*
* (C) Copyright 2023 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra.config.validation;
import com.google.common.base.Strings;
import org.nzbhydra.config.RestartRequired;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.lang.reflect.Field;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.List;
import java.util.Objects;
import java.util.regex.Pattern;
import java.util.regex.PatternSyntaxException;
public class ConfigValidationTools {
private static final Logger logger = LoggerFactory.getLogger(ConfigValidationTools.class);
static void checkRegex(List<String> errorMessages, String regex, String errorMessage) {
if (!Strings.isNullOrEmpty(regex)) {
try {
Pattern.compile(regex);
} catch (PatternSyntaxException e) {
errorMessages.add(errorMessage);
}
}
}
/**
* Detects if any setting was changed that requires a restart to be effective
*
* @param oldConfig the old config (its settings will be compared with the ones from the calling instance)
* @return
*/
public static boolean isRestartNeeded(Object oldConfig, Object newConfig) {
for (Field field : oldConfig.getClass().getDeclaredFields()) {
if (field.isAnnotationPresent(RestartRequired.class)) {
try {
//PropertyDescriptor doesn't work for some reason, this is just as fine for what we need
String getterName = (field.getType() == Boolean.class || field.getType() == boolean.class ? "is" : "get") + field.getName().substring(0, 1).toUpperCase() + field.getName().substring(1);
Method method = oldConfig.getClass().getDeclaredMethod(getterName);
Object oldValue = method.invoke(oldConfig);
Object newValue = method.invoke(newConfig);
if (!Objects.equals(oldValue, newValue)) {
logger.debug("Restart needed because field {} has changed", field.getName());
return true;
}
} catch (InvocationTargetException | NoSuchMethodException | IllegalAccessException e) {
logger.error("Unable to determine if field '{}' in class {} was changed", field.getName(), oldConfig.getClass().getName());
}
}
}
return false;
}
}

55
core/src/main/java/org/nzbhydra/config/validation/ConfigValidator.java Normal file
View File

@ -0,0 +1,55 @@
/*
* (C) Copyright 2023 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra.config.validation;
import org.nzbhydra.config.BaseConfig;
public interface ConfigValidator<T> {
boolean doesValidate(Class<?> clazz);
/**
* @param oldBaseConfig old config state (e.g. to compare what has changed)
* @param newBaseConfig
* @param newConfig the new config. Will always be the same object as the one on which the method was called
* @return a list of error messages or an empty list when everything is fine
*/
ConfigValidationResult validateConfig(BaseConfig oldBaseConfig, BaseConfig newBaseConfig, T newConfig);
/**
* Called before the config is saved after the user made some changes. Use this to convert data, e.g. passwords.
*
* @param oldBaseConfig
*/
default T prepareForSaving(BaseConfig oldBaseConfig, T newConfig) {
return newConfig;
}
/**
* Called before the config is transferred to the GUI. Use this to prepare data, e.g. passwords.
*/
default T updateAfterLoading(T newConfig) {
return newConfig;
}
/**
* Called for a new config to initialize itself
*/
default T initializeNewConfig(T newConfig) {
return newConfig;
}
}

72
core/src/main/java/org/nzbhydra/config/downloading/DownloaderConfig.java → core/src/main/java/org/nzbhydra/config/validation/DownloaderConfigValidator.java
View File

@ -1,5 +1,5 @@
/*
* (C) Copyright 2017 TheOtherP (theotherp@posteo.net)
* (C) Copyright 2023 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@ -14,62 +14,33 @@
* limitations under the License.
*/
package org.nzbhydra.config.downloading;
package org.nzbhydra.config.validation;
import com.google.common.base.Strings;
import lombok.Data;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.config.ValidatingConfig;
import org.nzbhydra.config.downloading.DownloaderConfig;
import org.nzbhydra.config.downloading.NzbAddingType;
import org.nzbhydra.config.indexer.IndexerConfig;
import org.nzbhydra.config.sensitive.SensitiveData;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Optional;
@Data
@ConfigurationProperties(prefix = "downloaders")
public class DownloaderConfig extends ValidatingConfig<DownloaderConfig> {
@SensitiveData
private String apiKey;
private String defaultCategory;
private DownloadType downloadType;
private boolean enabled;
private String iconCssClass;
private String name;
private NzbAddingType nzbAddingType;
private DownloaderType downloaderType;
@SensitiveData
private String url;
@SensitiveData
private String username;
@SensitiveData
private String password;
private boolean addPaused;
public DownloaderType getDownloaderType() {
return downloaderType;
}
public Optional<String> getUsername() {
return Optional.ofNullable(Strings.emptyToNull(username));
}
public Optional<String> getPassword() {
return Optional.ofNullable(Strings.emptyToNull(password));
@Component
public class DownloaderConfigValidator implements ConfigValidator<DownloaderConfig> {
@Override
public boolean doesValidate(Class<?> clazz) {
return clazz == DownloaderConfig.class;
}
@Override
public ConfigValidationResult validateConfig(BaseConfig oldConfig, DownloaderConfig newDownloaderConfig, BaseConfig newBaseConfig) {
public ConfigValidationResult validateConfig(BaseConfig oldBaseConfig, BaseConfig newBaseConfig, DownloaderConfig newConfig) {
List<String> warnings = new ArrayList<>();
if (isEnabledWithoutSendLink(newBaseConfig, "nzbs.in", newDownloaderConfig)) {
if (isEnabledWithoutSendLink(newBaseConfig, "nzbs.in", newConfig)) {
warnings.add("nzbs.in forbids NZBHydra to download NZBs directly. The NZB adding type \"Send link\" will automatically used for this indexer.");
}
if (isEnabledWithoutSendLink(newBaseConfig, "omgwtfnzbs", newDownloaderConfig)) {
if (isEnabledWithoutSendLink(newBaseConfig, "omgwtfnzbs", newConfig)) {
warnings.add("omgwtfnzbs forbids NZBHydra to download NZBs directly. The NZB adding type \"Send link\" will automatically used for this indexer.");
}
return new ConfigValidationResult(true, false, Collections.emptyList(), warnings);
@ -78,21 +49,4 @@ public class DownloaderConfig extends ValidatingConfig<DownloaderConfig> {
private static boolean isEnabledWithoutSendLink(BaseConfig newBaseConfig, String hostContains, DownloaderConfig newDownloaderConfig) {
return newBaseConfig.getIndexers().stream().anyMatch(x -> x.getHost().toLowerCase().contains(hostContains) && x.getState() == IndexerConfig.State.ENABLED) && newDownloaderConfig.getNzbAddingType() != NzbAddingType.SEND_LINK;
}
@Override
public DownloaderConfig prepareForSaving(BaseConfig oldBaseConfig) {
return this;
}
@Override
public DownloaderConfig updateAfterLoading() {
return this;
}
@Override
public DownloaderConfig initializeNewConfig() {
return this;
}
}

88
core/src/main/java/org/nzbhydra/config/downloading/DownloadingConfig.java → core/src/main/java/org/nzbhydra/config/validation/DownloadingConfigValidator.java
View File

@ -1,5 +1,5 @@
/*
* (C) Copyright 2017 TheOtherP (theotherp@posteo.net)
* (C) Copyright 2023 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@ -14,55 +14,44 @@
* limitations under the License.
*/
package org.nzbhydra.config.downloading;
package org.nzbhydra.config.validation;
import com.fasterxml.jackson.annotation.JsonFormat;
import com.google.common.base.Strings;
import lombok.Data;
import org.javers.core.metamodel.annotation.DiffIgnore;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.config.SearchSourceRestriction;
import org.nzbhydra.config.ValidatingConfig;
import org.nzbhydra.config.downloading.DownloadingConfig;
import org.nzbhydra.config.downloading.FileDownloadAccessType;
import org.nzbhydra.config.indexer.IndexerConfig;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import java.io.File;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.Optional;
import java.util.stream.Collectors;
@Data
@ConfigurationProperties(prefix = "downloading")
public class DownloadingConfig extends ValidatingConfig<DownloadingConfig> {
@Component
public class DownloadingConfigValidator implements ConfigValidator<DownloadingConfig> {
@DiffIgnore
private List<DownloaderConfig> downloaders = new ArrayList<>();
private String saveTorrentsTo;
private String saveNzbsTo;
private boolean sendMagnetLinks;
private boolean updateStatuses;
private boolean showDownloaderStatus = true;
@JsonFormat(shape = JsonFormat.Shape.STRING)
private FileDownloadAccessType nzbAccessType = FileDownloadAccessType.REDIRECT;
private SearchSourceRestriction fallbackForFailed = SearchSourceRestriction.BOTH;
private String externalUrl;
private String primaryDownloader;
@Autowired
private DownloaderConfigValidator downloaderConfigValidator;
@Override
public ConfigValidationResult validateConfig(BaseConfig oldConfig, DownloadingConfig newConfig, BaseConfig newBaseConfig) {
public boolean doesValidate(Class<?> clazz) {
return clazz == DownloadingConfig.class;
}
@Override
public ConfigValidationResult validateConfig(BaseConfig oldBaseConfig, BaseConfig newBaseConfig, DownloadingConfig newConfig) {
List<String> errors = new ArrayList<>();
if (getSaveTorrentsTo().isPresent()) {
File file = new File(getSaveTorrentsTo().get());
validateBlackholeFolder(errors, file, getSaveTorrentsTo().get(), "Torrent");
if (newConfig.getSaveTorrentsTo().isPresent()) {
File file = new File(newConfig.getSaveTorrentsTo().get());
validateBlackholeFolder(errors, file, newConfig.getSaveTorrentsTo().get(), "Torrent");
}
if (getSaveNzbsTo().isPresent()) {
File file = new File(getSaveNzbsTo().get());
validateBlackholeFolder(errors, file, getSaveNzbsTo().get(), "NZB");
if (newConfig.getSaveNzbsTo().isPresent()) {
File file = new File(newConfig.getSaveNzbsTo().get());
validateBlackholeFolder(errors, file, newConfig.getSaveNzbsTo().get(), "NZB");
}
List<ConfigValidationResult> validationResults = downloaders.stream().map(downloaderConfig -> downloaderConfig.validateConfig(oldConfig, downloaderConfig, newBaseConfig)).collect(Collectors.toList());
List<String> downloaderErrors = validationResults.stream().map(ConfigValidationResult::getErrorMessages).flatMap(Collection::stream).collect(Collectors.toList());
List<ConfigValidationResult> validationResults = newConfig.getDownloaders().stream().map(downloaderConfig -> downloaderConfigValidator.validateConfig(oldBaseConfig, newBaseConfig, downloaderConfig)).toList();
List<String> downloaderErrors = validationResults.stream().map(ConfigValidationResult::getErrorMessages).flatMap(Collection::stream).toList();
errors.addAll(downloaderErrors);
List<String> warnings = new ArrayList<>();
@ -74,7 +63,7 @@ public class DownloadingConfig extends ValidatingConfig<DownloadingConfig> {
warnings.add("omgwftnzbs forbids NZBHydra to download NZBs directly. The NZB access type \"Redirect to indexer\" will automatically be used for this indexer.");
}
warnings.addAll(validationResults.stream().map(ConfigValidationResult::getWarningMessages).flatMap(Collection::stream).collect(Collectors.toList()));
warnings.addAll(validationResults.stream().map(ConfigValidationResult::getWarningMessages).flatMap(Collection::stream).toList());
return new ConfigValidationResult(errors.isEmpty(), false, errors, warnings);
}
@ -98,31 +87,4 @@ public class DownloadingConfig extends ValidatingConfig<DownloadingConfig> {
}
}
public Optional<String> getSaveTorrentsTo() {
return Optional.ofNullable(Strings.emptyToNull(saveTorrentsTo));
}
public Optional<String> getSaveNzbsTo() {
return Optional.ofNullable(saveNzbsTo);
}
public Optional<String> getExternalUrl() {
return Optional.ofNullable(externalUrl);
}
@Override
public DownloadingConfig prepareForSaving(BaseConfig oldBaseConfig) {
return this;
}
@Override
public DownloadingConfig updateAfterLoading() {
return this;
}
@Override
public DownloadingConfig initializeNewConfig() {
return this;
}
}

79
core/src/main/java/org/nzbhydra/config/validation/IndexerConfigValidator.java Normal file
View File

@ -0,0 +1,79 @@
/*
* (C) Copyright 2023 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra.config.validation;
import com.google.common.base.Strings;
import org.apache.commons.lang3.StringUtils;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.config.indexer.IndexerConfig;
import org.nzbhydra.searching.IndexerForSearchSelector;
import org.springframework.stereotype.Component;
import java.time.format.DateTimeFormatter;
import java.util.regex.Matcher;
@Component
public class IndexerConfigValidator implements ConfigValidator<IndexerConfig> {
@Override
public boolean doesValidate(Class<?> clazz) {
return clazz == IndexerConfig.class;
}
@Override
public ConfigValidationResult validateConfig(BaseConfig oldBaseConfig, BaseConfig newBaseConfig, IndexerConfig newConfig) {
ConfigValidationResult validationResult = new ConfigValidationResult();
for (String schedule : newConfig.getSchedule()) {
Matcher matcher = IndexerForSearchSelector.SCHEDULER_PATTERN.matcher(schedule);
if (!matcher.matches()) {
validationResult.getErrorMessages().add("Indexer " + newConfig.getName() + " contains an invalid schedule: " + schedule);
}
}
if (newConfig.getHitLimit().isPresent() && newConfig.getHitLimit().get() <= 0) {
validationResult.getErrorMessages().add("Indexer " + newConfig.getName() + " has a hit limit of 0 or lower which doesn't make sense: ");
}
if (newConfig.getDownloadLimit().isPresent() && newConfig.getDownloadLimit().get() <= 0) {
validationResult.getErrorMessages().add("Indexer " + newConfig.getName() + " has a download limit of 0 or lower which doesn't make sense: ");
}
final String newExpirationDate = newConfig.getVipExpirationDate();
if (newExpirationDate != null && !newExpirationDate.equals("Lifetime")) {
try {
DateTimeFormatter.ofPattern("yyyy-MM-dd").parse(newExpirationDate);
} catch (Exception e) {
validationResult.getErrorMessages().add("Invalid expiry date for indexer " + newConfig.getName() + ". Either use 'Lifetime' or use the format `YYYY-MM-DD");
}
}
newConfig.getCustomParameters().forEach(x -> {
if (Strings.isNullOrEmpty(x) || StringUtils.countMatches(x, '=') > 1) {
validationResult.getErrorMessages().add("The custom paramater " + x + " is invalid. You must use the format name=value.");
}
});
return validationResult;
}
@Override
public IndexerConfig prepareForSaving(BaseConfig oldBaseConfig, IndexerConfig newConfig) {
if (newConfig.getState() == IndexerConfig.State.ENABLED || newConfig.getState() == IndexerConfig.State.DISABLED_USER) {
newConfig.setDisabledUntil(null);
newConfig.setDisabledLevel(0);
newConfig.setLastError(null);
}
return newConfig;
}
}

63
core/src/main/java/org/nzbhydra/config/validation/LoggingConfigValidator.java Normal file
View File

@ -0,0 +1,63 @@
/*
* (C) Copyright 2023 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra.config.validation;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.config.LoggingConfig;
import org.nzbhydra.logging.LoggingMarkers;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
import java.util.Arrays;
import java.util.Iterator;
@Component
public class LoggingConfigValidator implements ConfigValidator<LoggingConfig> {
private static final Logger logger = LoggerFactory.getLogger(LoggingConfigValidator.class);
@Override
public boolean doesValidate(Class<?> clazz) {
return clazz == LoggingConfig.class;
}
@Override
public ConfigValidationResult validateConfig(BaseConfig oldBaseConfig, BaseConfig newBaseConfig, LoggingConfig newConfig) {
ConfigValidationResult result = new ConfigValidationResult();
result.setRestartNeeded(ConfigValidationTools.isRestartNeeded(oldBaseConfig.getMain().getLogging(), newConfig));
if (newBaseConfig.getMain().getLogging().getMarkersToLog().size() > 3) {
result.getWarningMessages().add("You have more than 3 logging markers enabled. This is very rarely useful. Please make sure that this is actually needed. When creating debug infos please only enable those markers requested by the developer.");
}
return result;
}
@Override
public LoggingConfig prepareForSaving(BaseConfig oldBaseConfig, LoggingConfig newConfig) {
for (Iterator<String> iterator = newConfig.getMarkersToLog().iterator(); iterator.hasNext(); ) {
String marker = iterator.next();
if (Arrays.stream(LoggingMarkers.class.getDeclaredFields()).noneMatch(x -> x.getName().equals(marker))) {
logger.info("Removing logging marker that doesn't exist anymore.");
iterator.remove();
}
}
return newConfig;
}
}

139
core/src/main/java/org/nzbhydra/config/validation/MainConfigValidator.java Normal file
View File

@ -0,0 +1,139 @@
/*
* (C) Copyright 2023 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra.config.validation;
import com.google.common.base.Strings;
import org.nzbhydra.NzbHydra;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.config.MainConfig;
import org.nzbhydra.debuginfos.DebugInfosProvider;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import java.io.File;
import java.io.IOException;
import java.math.BigInteger;
import java.net.InetAddress;
import java.util.Random;
import static org.nzbhydra.config.validation.ConfigValidationTools.isRestartNeeded;
@Component
public class MainConfigValidator implements ConfigValidator<MainConfig> {
@Autowired
private LoggingConfigValidator loggingConfigValidator;
@Override
public boolean doesValidate(Class<?> clazz) {
return clazz == MainConfig.class;
}
@Override
public ConfigValidationResult validateConfig(BaseConfig oldBaseConfig, BaseConfig newBaseConfig, MainConfig newConfig) {
ConfigValidationResult result = new ConfigValidationResult();
MainConfig oldMain = oldBaseConfig.getMain();
boolean portChanged = oldMain.getPort() != newConfig.getPort();
boolean urlBaseChanged = oldMain.getUrlBase().isPresent() && !oldMain.getUrlBase().get().equals(newConfig.getUrlBase().orElse(null));
if (newConfig.getUrlBase().isEmpty() && oldMain.getUrlBase().isPresent() && oldMain.getUrlBase().get().equals("/")) {
urlBaseChanged = false;
}
boolean sslChanged = oldMain.isSsl() != newConfig.isSsl();
if (portChanged || urlBaseChanged || sslChanged && !newConfig.isStartupBrowser()) {
result.getWarningMessages().add("You've made changes that affect Hydra's URL and require a restart. Hydra will try and reload using the new URL when it's back.");
}
if (DebugInfosProvider.isRunInDocker() && !"0.0.0.0".equals(newConfig.getHost())) {
result.getWarningMessages().add("You've changed the host but NZBHydra seems to be run in docker. It's recommended to use the host '0.0.0.0'.");
}
if (!"0.0.0.0".equals(newConfig.getHost())) {
try {
boolean reachable = InetAddress.getByName(newConfig.getHost()).isReachable(1);
if (!reachable) {
result.getWarningMessages().add("The configured host address cannot be reached. Are you sure it is correct?");
}
} catch (IOException e) {
//Ignore, user will have to know what he does
}
}
if (oldBaseConfig.getMain().getXmx() < 128) {
result.getErrorMessages().add("The JVM memory must be set to at least 128");
}
MainConfig newMain = newBaseConfig.getMain();
if (newMain.getKeepHistoryForWeeks() != null && newMain.getKeepHistoryForWeeks() <= 0) {
result.getErrorMessages().add("Please either delete the value for \"Keep history for\" or set it to a positive value.");
}
if (newMain.getKeepStatsForWeeks() != null && newMain.getKeepStatsForWeeks() <= 0) {
result.getErrorMessages().add("Please either delete the value for \"Keep stats for\" or set it to a positive value.");
}
if (newMain.getKeepStatsForWeeks() != null && newMain.getKeepHistoryForWeeks() != null && newMain.getKeepStatsForWeeks() > newMain.getKeepHistoryForWeeks()) {
result.getErrorMessages().add("Please set the time to keep stats to a value not higher than the time to keep history.");
}
if (newMain.getBackupFolder() != null) {
final File backupFolderFile;
if (newConfig.getBackupFolder().contains(File.separator)) {
backupFolderFile = new File(newConfig.getBackupFolder());
} else {
backupFolderFile = new File(NzbHydra.getDataFolder(), newConfig.getBackupFolder());
}
if (!backupFolderFile.exists()) {
final boolean created = backupFolderFile.mkdirs();
if (!created) {
result.getErrorMessages().add("Backup folder " + newConfig.getBackupFolder() + " does not exist and could not be created");
}
}
}
ConfigValidationResult validationResult = loggingConfigValidator.validateConfig(oldBaseConfig, newBaseConfig, newConfig.getLogging());
result.getWarningMessages().addAll(validationResult.getWarningMessages());
result.getErrorMessages().addAll(validationResult.getErrorMessages());
oldMain = prepareForSaving(oldBaseConfig, oldMain);
result.setRestartNeeded(validationResult.isRestartNeeded() || isRestartNeeded(oldMain, newConfig));
result.setOk(validationResult.isOk() && result.isOk());
return result;
}
@Override
public MainConfig prepareForSaving(BaseConfig oldBaseConfig, MainConfig newConfig) {
final String urlBase = newConfig.getUrlBase().orElse(null);
if (!Strings.isNullOrEmpty(urlBase) && (!urlBase.startsWith("/") || urlBase.endsWith("/") || "/".equals(urlBase))) {
if (!urlBase.startsWith("/")) {
newConfig.setUrlBase("/" + urlBase);
}
if (urlBase.endsWith("/")) {
newConfig.setUrlBase(urlBase.substring(0, urlBase.length() - 1));
}
if ("/".equals(urlBase) || "".equals(urlBase)) {
newConfig.setUrlBase("/");
}
newConfig.setUrlBase(urlBase);
}
return newConfig;
}
@Override
public MainConfig initializeNewConfig(MainConfig newConfig) {
Random random = new Random();
newConfig.setApiKey(new BigInteger(130, random).toString(32).toUpperCase());
return newConfig;
}
}

52
core/src/main/java/org/nzbhydra/config/validation/NotificationConfigValidator.java Normal file
View File

@ -0,0 +1,52 @@
/*
* (C) Copyright 2023 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra.config.validation;
import joptsimple.internal.Strings;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.config.NotificationConfig;
import org.springframework.stereotype.Component;
import java.util.ArrayList;
import java.util.List;
@Component
public class NotificationConfigValidator implements ConfigValidator<NotificationConfig> {
@Override
public boolean doesValidate(Class<?> clazz) {
return clazz == NotificationConfig.class;
}
@Override
public ConfigValidationResult validateConfig(BaseConfig oldBaseConfig, BaseConfig newBaseConfig, NotificationConfig newConfig) {
final List<String> errors = new ArrayList<>();
final List<String> warnings = new ArrayList<>();
if (newConfig.getEntries().stream()
.anyMatch(x -> Strings.isNullOrEmpty(x.getAppriseUrls()))) {
errors.add("Make sure all notification entries contain a URL");
}
final boolean appriseUrlSet = !Strings.isNullOrEmpty(newConfig.getAppriseApiUrl());
final boolean anyEntries = newConfig.getEntries().isEmpty();
if (anyEntries && !appriseUrlSet) {
warnings.add("No notifications will be sent unless the Apprise API URL is configured.");
}
return new ConfigValidationResult(true, false, errors, warnings);
}
}

102
core/src/main/java/org/nzbhydra/config/validation/SearchingConfigValidator.java Normal file
View File

@ -0,0 +1,102 @@
/*
* (C) Copyright 2023 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra.config.validation;
import com.google.common.base.Strings;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.config.SearchSourceRestriction;
import org.nzbhydra.config.SearchingConfig;
import org.nzbhydra.config.searching.CustomQueryAndTitleMapping;
import org.nzbhydra.searching.CustomQueryAndTitleMappingHandler;
import org.nzbhydra.searching.searchrequests.SearchRequest;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;
import static org.nzbhydra.config.validation.ConfigValidationTools.checkRegex;
@Component
public class SearchingConfigValidator implements ConfigValidator<SearchingConfig> {
private static final Logger logger = LoggerFactory.getLogger(SearchingConfigValidator.class);
@Override
public boolean doesValidate(Class<?> clazz) {
return clazz == SearchingConfig.class;
}
@Override
public ConfigValidationResult validateConfig(BaseConfig oldBaseConfig, BaseConfig newBaseConfig, SearchingConfig newConfig) {
List<String> errors = new ArrayList<>();
List<String> warnings = new ArrayList<>();
checkRegex(errors, newConfig.getRequiredRegex().orElse(null), "The required regex in \"Searching\" is invalid");
checkRegex(errors, newConfig.getForbiddenRegex().orElse(null), "The forbidden in \"Searching\" is invalid");
if (newConfig.getApplyRestrictions() == SearchSourceRestriction.NONE) {
if (!newConfig.getRequiredWords().isEmpty() || !newConfig.getForbiddenWords().isEmpty()) {
warnings.add("You selected not to apply any word restrictions in \"Searching\" but supplied forbidden or required words there");
}
if (newConfig.getRequiredRegex().isPresent() || newConfig.getForbiddenRegex().isPresent()) {
warnings.add("You selected not to apply any word restrictions in \"Searching\" but supplied a forbidden or required regex there");
}
}
final CustomQueryAndTitleMappingHandler customQueryAndTitleMappingHandler = new CustomQueryAndTitleMappingHandler(newBaseConfig);
final SearchRequest searchRequest = new SearchRequest();
searchRequest.setTitle("test title");
searchRequest.setQuery("test query");
for (CustomQueryAndTitleMapping customCustomQueryAndTitleMapping : newConfig.getCustomMappings()) {
try {
customQueryAndTitleMappingHandler.mapSearchRequest(searchRequest, Collections.singletonList(customCustomQueryAndTitleMapping));
} catch (Exception e) {
errors.add(String.format("Unable to process mapping %s:}\n%s", customCustomQueryAndTitleMapping.toString(), e.getMessage()));
}
if (customCustomQueryAndTitleMapping.getFrom().contains("{episode:")) {
errors.add("The group 'episode' is not allowed in custom mapping input patterns.");
}
if (customCustomQueryAndTitleMapping.getFrom().contains("{season:")) {
errors.add("The group 'season' is not allowed in custom mapping input patterns.");
}
}
final List<String> emptyTrailing = (newConfig.getRemoveTrailing().stream().filter(Strings::isNullOrEmpty)).toList();
if (!emptyTrailing.isEmpty()) {
errors.add("Trailing values to remove contains empty values");
}
return new ConfigValidationResult(errors.isEmpty(), false, errors, warnings);
}
@Override
public SearchingConfig prepareForSaving(BaseConfig oldBaseConfig, SearchingConfig newConfig) {
final Set<String> customQuickfilterNames = newConfig.getCustomQuickFilterButtons().stream().map(x -> x.split("=")[0]).collect(Collectors.toSet());
for (Iterator<String> iterator = newConfig.getPreselectQuickFilterButtons().iterator(); iterator.hasNext(); ) {
String preselectQuickFilterButton = iterator.next();
final String[] split = preselectQuickFilterButton.split("\\|");
if ("custom".equals(split[0]) && !customQuickfilterNames.contains(split[0])) {
logger.info("Custom quickfilter {} doesn't exist anymore, removing it from list of filters to preselect.", preselectQuickFilterButton);
iterator.remove();
}
}
return newConfig;
}
}

51
core/src/main/java/org/nzbhydra/config/validation/UserAuthConfigValidator.java Normal file
View File

@ -0,0 +1,51 @@
/*
* (C) Copyright 2023 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra.config.validation;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.config.auth.UserAuthConfig;
import org.springframework.stereotype.Component;
@Component
public class UserAuthConfigValidator implements ConfigValidator<UserAuthConfig> {
@Override
public boolean doesValidate(Class<?> clazz) {
return clazz == UserAuthConfig.class;
}
@Override
public ConfigValidationResult validateConfig(BaseConfig oldBaseConfig, BaseConfig newBaseConfig, UserAuthConfig newConfig) {
return new ConfigValidationResult();
}
@Override
public UserAuthConfig prepareForSaving(BaseConfig oldBaseConfig, UserAuthConfig newConfig) {
if (newConfig.getPassword() != null && !newConfig.getPassword().startsWith(UserAuthConfig.PASSWORD_ID)) {
newConfig.setPassword(UserAuthConfig.PASSWORD_ID + newConfig.getPassword());
}
return newConfig;
}
@Override
public UserAuthConfig updateAfterLoading(UserAuthConfig newConfig) {
if (newConfig.getPassword() != null && newConfig.getPassword().startsWith(UserAuthConfig.PASSWORD_ID)) {
newConfig.setPassword(newConfig.getPassword().substring(6));
}
return newConfig;
}
}

266
core/src/main/java/org/nzbhydra/database/DatabaseRecreation.java
View File

@ -16,179 +16,223 @@
package org.nzbhydra.database;
import com.google.common.collect.Sets;
import com.google.common.base.Joiner;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.EqualsAndHashCode;
import org.apache.commons.io.IOUtils;
import org.flywaydb.core.internal.resource.StringResource;
import org.apache.commons.io.filefilter.WildcardFileFilter;
import org.flywaydb.core.Flyway;
import org.nzbhydra.NzbHydra;
import org.nzbhydra.springnative.ReflectionMarker;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.core.io.Resource;
import org.springframework.core.io.support.PathMatchingResourcePatternResolver;
import org.springframework.http.HttpMethod;
import org.springframework.http.HttpStatus;
import org.springframework.http.client.ClientHttpRequest;
import org.springframework.http.client.ClientHttpResponse;
import org.springframework.http.client.OkHttp3ClientHttpRequestFactory;
import java.io.File;
import java.io.FileWriter;
import java.io.FileReader;
import java.io.FilenameFilter;
import java.io.IOException;
import java.nio.charset.Charset;
import java.io.InputStream;
import java.net.URI;
import java.nio.file.Files;
import java.nio.file.StandardCopyOption;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.HashSet;
import java.util.Arrays;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
@SuppressWarnings("ResultOfMethodCallIgnored")
public class DatabaseRecreation {
private static final Logger logger = LoggerFactory.getLogger(DatabaseRecreation.class);
private static final Map<String, String> SCHEMA_VERSION_CHANGES = new LinkedHashMap<>();
static {
SCHEMA_VERSION_CHANGES.put("'V1.11__REMOVE_INDEXERSTATUSES.sql', \\-?\\d+", "'V1.11__REMOVE_INDEXERSTATUSES.sql', 898390205");
SCHEMA_VERSION_CHANGES.put("'V1.17__SHORTACCESS_AGAIN.sql', \\-?\\d+", "'V1.17__SHORTACCESS_AGAIN.sql', 177884391");
}
public static void runDatabaseScript() throws ClassNotFoundException, SQLException {
if (!Thread.currentThread().getName().equals("main")) {
//During development this class is called twice (because of the Spring developer tools)
logger.debug("Skipping database script check for thread {}", Thread.currentThread().getName());
return;
}
public static void runDatabaseScript() throws Exception {
File databaseFile = new File(NzbHydra.getDataFolder(), "database/nzbhydra.mv.db");
File databaseScriptFile = new File(NzbHydra.getDataFolder(), "databaseScript.sql");
File databaseScriptFileNew = new File(NzbHydra.getDataFolder(), "databaseScriptNew.sql");
File restoreScriptFile = new File(NzbHydra.getDataFolder(), "database/script.sql");
String dbConnectionUrl = "jdbc:h2:file:" + databaseFile.getAbsolutePath().replace(".mv.db", "");
Class.forName("org.h2.Driver");
migrateToH2v2IfNeeded(databaseFile, dbConnectionUrl);
if (restoreScriptFile.exists() && !databaseScriptFile.exists()) {
DatabaseRecreation.logger.info("No database file found but script.sql - restoring database");
try (Connection connection = DriverManager.getConnection(dbConnectionUrl, "sa", "sa")) {
connection.createStatement().executeUpdate("runscript from '%s';".formatted(restoreScriptFile.getCanonicalPath().replace("\\", "/")));
restoreScriptFile.delete();
}
}
}
@SuppressWarnings("ResultOfMethodCallIgnored")
private static void migrateToH2v2IfNeeded(File databaseFile, String dbConnectionUrl) throws Exception {
if (!databaseFile.exists()) {
logger.debug("No database file found - no recreation needed");
return;
}
Class.forName("org.h2.Driver");
String dbConnectionUrl = "jdbc:h2:file:" + databaseFile.getAbsolutePath().replace(".mv.db", "");
if (isDatabaseRecreationNotNeeded(dbConnectionUrl)) {
try {
char[] buffer;
try (FileReader fileReader = new FileReader(databaseFile)) {
buffer = new char[1024];
final int read = fileReader.read(buffer);
}
final String header = new String(buffer).trim();
if (header.contains("format:1")) {
logger.info("Determined existing database to be version 1.4. Migration needed.");
} else if (header.contains("format:2")) {
logger.info("Determined existing database to be version 2. No migration needed.");
return;
} else {
logger.error("Unable to determine database version from header {}", header);
throw new RuntimeException("Invalid database file header");
}
createDatabaseScript(databaseScriptFile, dbConnectionUrl);
deleteExistingDatabase(databaseFile);
replaceSchemaVersionChanges(databaseScriptFile, databaseScriptFileNew);
runDatabaseScript(databaseScriptFile, dbConnectionUrl);
} catch (Exception e) {
throw new RuntimeException("Unable to open database file " + databaseFile, e);
}
boolean isUpgrade14To210 = true;
if (isUpgrade14To210) {
private static boolean isDatabaseRecreationNotNeeded(String dbConnectionUrl) throws SQLException {
logger.debug("Determining if database recreation is needed");
Set<ExecutedScript> executedScripts = new HashSet<>();
try (Connection conn = DriverManager.getConnection(dbConnectionUrl, "SA", "")) {
ResultSet resultSet = conn.createStatement().executeQuery("select \"script\", \"checksum\" from \"schema_version\";");
while (resultSet.next()) {
executedScripts.add(new ExecutedScript(resultSet.getString(1), resultSet.getInt((2))));
}
}
PathMatchingResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
try {
HashSet<Resource> resources = Sets.newHashSet(resolver.getResources("classpath:/migration/*"));
if (resources.stream().allMatch(x -> {
try {
StringResource stringResource = new StringResource(IOUtils.toString(x.getInputStream(), Charset.defaultCharset()));
ExecutedScript executedScript = new ExecutedScript(x.getFilename(), stringResource.checksum());
boolean scriptExecuted = executedScripts.contains(executedScript);
if (!scriptExecuted) {
logger.info("Database recreation needed because {} was not yet executed or its checksum has changed", executedScript);
final File[] traceFiles = databaseFile.getParentFile().listFiles((FilenameFilter) new WildcardFileFilter("*.trace.db"));
if (traceFiles != null) {
for (File traceFile : traceFiles) {
traceFile.delete();
}
return scriptExecuted;
} catch (IOException e) {
throw new RuntimeException("Unable to determine checksum for " + x.getFilename());
}
})) {
logger.debug("No migration scripts found to run. Skipping database recreation");
return true;
}
} catch (IOException e) {
throw new RuntimeException("Unable to find migration scripts", e);
}
return false;
} catch (Exception e) {
logger.error("Unable to delete trace files", e);
}
private static void runDatabaseScript(File databaseScriptFile, String dbConnectionUrl) throws SQLException {
try (Connection conn = DriverManager.getConnection(dbConnectionUrl, "SA", "")) {
logger.info("Running database script {} for reimport of old database", databaseScriptFile.getAbsolutePath());
File backupDatabaseFile = null;
String javaExecutable;
final File h2OldJar;
final File h2NewJar;
final String scriptFilePath;
try {
conn.createStatement().execute("runscript from '" + databaseScriptFile.getAbsolutePath() + "'");
} catch (SQLException e) {
throw new RuntimeException("Unable to import database script", e);
javaExecutable = getJavaExecutable();
h2OldJar = downloadJarFile("https://repo1.maven.org/maven2/com/h2database/h2/1.4.200/h2-1.4.200.jar");
h2NewJar = downloadJarFile("https://repo1.maven.org/maven2/com/h2database/h2/2.1.214/h2-2.1.214.jar");
} catch (Exception e) {
logger.error("Error migrating old database. Unable to download h2 jars");
throw e;
}
logger.info("Successfully recreated database");
try {
final File scriptFile = Files.createTempFile("nzbhydra", ".sql").toFile();
scriptFile.deleteOnExit();
scriptFilePath = scriptFile.getCanonicalPath();
logger.info("Running database migration from 1.4 to 2");
backupDatabaseFile = new File(databaseFile.getParent(), databaseFile.getName() + ".old.bak." + System.currentTimeMillis());
logger.info("Copying old database file {} to backup {} which will be automatically deleted after 14 days", databaseFile, backupDatabaseFile);
Files.copy(databaseFile.toPath(), backupDatabaseFile.toPath());
final String updatePasswordQuery = "alter user sa set password 'sa'";
updatePassword(dbConnectionUrl, javaExecutable, h2OldJar, updatePasswordQuery);
runH2Command(Arrays.asList(javaExecutable, "-cp", h2OldJar.toString(), "org.h2.tools.Script", "-url", dbConnectionUrl, "-user", "sa", "-password", "sa", "-script", scriptFilePath), "Database export failed.");
} catch (Exception e) {
logger.error("Error migrating old database file to new one");
if (backupDatabaseFile != null && backupDatabaseFile.exists()) {
if (backupDatabaseFile != null && backupDatabaseFile.exists()) {
backupDatabaseFile.delete();
}
boolean deleted = databaseScriptFile.delete();
}
throw e;
}
try {
final boolean deleted = databaseFile.delete();
if (!deleted) {
throw new RuntimeException("Unable to delete database script file at " + databaseScriptFile.getAbsolutePath() + ". Please delete it manually.");
throw new RuntimeException("Unable to delete old database file " + databaseFile);
}
runH2Command(Arrays.asList(javaExecutable, "-cp", h2NewJar.toString(), "org.h2.tools.RunScript", "-url", dbConnectionUrl, "-user", "sa", "-password", "sa", "-script", scriptFilePath, "-options", "FROM_1X"), "Database import failed.");
final Flyway flyway = Flyway.configure()
.dataSource(dbConnectionUrl, "sa", "sa")
.baselineDescription("INITIAL")
.baselineVersion("1")
.load();
flyway.baseline();
} catch (Exception e) {
logger.error("Error while trying to migrate database to 2.0");
if (backupDatabaseFile != null && backupDatabaseFile.exists()) {
logger.info("Restoring database file {} from backup {}", databaseFile, backupDatabaseFile);
Files.move(backupDatabaseFile.toPath(), databaseFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
}
throw new RuntimeException(e);
}
}
}
private static void replaceSchemaVersionChanges(File databaseScriptFile, File databaseScriptFileNew) {
private static void updatePassword(String dbConnectionUrl, String javaExecutable, File h2OldJar, String updatePasswordQuery) throws IOException, InterruptedException {
try {
try (FileWriter fileWriter = new FileWriter(databaseScriptFileNew)) {
Files.lines(databaseScriptFile.toPath()).forEach(line -> {
String sql = line;
for (Map.Entry<String, String> entry : SCHEMA_VERSION_CHANGES.entrySet()) {
sql = sql.replaceAll(entry.getKey(), entry.getValue());
}
try {
fileWriter.write(sql);
fileWriter.write(System.getProperty("line.separator"));
} catch (IOException e) {
throw new RuntimeException("Unable to write to temp file " + databaseScriptFileNew, e);
}
});
}
if (!databaseScriptFile.delete()) {
throw new RuntimeException("Unable to delete existing database script file " + databaseScriptFile);
}
Files.move(databaseScriptFileNew.toPath(), databaseScriptFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
} catch (IOException e) {
throw new RuntimeException("Unable to update database migration versions", e);
runH2Command(Arrays.asList(javaExecutable, "-cp", h2OldJar.toString(), "org.h2.tools.Shell", "-url", dbConnectionUrl, "-user", "sa", "-sql", NzbHydra.isOsWindows() ? ("\"" + updatePasswordQuery + "\"") : updatePasswordQuery), "Password update failed.");
} catch (Exception e) {
runH2Command(Arrays.asList(javaExecutable, "-cp", h2OldJar.toString(), "org.h2.tools.Shell", "-url", dbConnectionUrl, "-user", "sa", "-password", "sa", "-sql", NzbHydra.isOsWindows() ? ("\"" + updatePasswordQuery + "\"") : updatePasswordQuery), "Password update failed.");
}
}
private static void deleteExistingDatabase(File databaseFile) {
if (!databaseFile.exists()) {
throw new RuntimeException("Unable to find database file at " + databaseFile.getAbsolutePath());
}
boolean deleted = databaseFile.delete();
if (!deleted) {
throw new RuntimeException("Unable to delete database file at " + databaseFile.getAbsolutePath() + ". Please move it somewhere else (just to be sure) and restart NZBHYdra.");
private static void runH2Command(List<String> updatePassCommand, String errorMessage) throws IOException, InterruptedException {
logger.info("Running command: " + Joiner.on(" ").join(updatePassCommand));
final Process process = new ProcessBuilder(updatePassCommand)
.redirectErrorStream(true)
.inheritIO()
.start();
final int result = process.waitFor();
if (result != 0) {
throw new RuntimeException(errorMessage + ". Code: " + result);
}
}
private static void createDatabaseScript(File databaseScriptFile, String url) throws SQLException {
if (databaseScriptFile.exists()) {
boolean deleted = databaseScriptFile.delete();
if (!deleted) {
throw new RuntimeException("Unable to delete database script file at " + databaseScriptFile.getAbsolutePath() + ". Please delete it manually and restart NZBHYdra.");
private static File downloadJarFile(String url) throws IOException {
final ClientHttpRequest request = new OkHttp3ClientHttpRequestFactory().createRequest(URI.create(url), HttpMethod.GET);
final File jarFile;
try (ClientHttpResponse response = request.execute()) {
jarFile = Files.createTempFile("nzbhydra", ".jar").toFile();
logger.debug("Downloaded file from {} to {}. Will be deleted on exit", url, jarFile);
jarFile.deleteOnExit();
try (InputStream body = response.getBody()) {
com.google.common.io.Files.asByteSink(jarFile).writeFrom(body);
}
if (response.getStatusCode() != HttpStatus.OK) {
throw new RuntimeException("Unable to download database library. Response: " + response.getStatusCode());
}
}
logger.info("Recreating database to ensure successful migration. This may take a couple of minutes...");
return jarFile;
try (Connection conn = DriverManager.getConnection(url, "SA", "")) {
logger.info("Creating database script {} from database", databaseScriptFile.getAbsolutePath());
conn.createStatement().execute(String.format("script to '%s'", databaseScriptFile));
if (!databaseScriptFile.exists()) {
throw new RuntimeException("Database script file was not created at " + databaseScriptFile.getAbsolutePath());
}
private static String getJavaExecutable() {
String javaExecutable;
if (System.getProperty("os.name").startsWith("Win")) {
javaExecutable = System.getProperties().getProperty("java.home") + File.separator + "bin" + File.separator + "java.exe";
} else {
javaExecutable = System.getProperties().getProperty("java.home") + File.separator + "bin" + File.separator + "java";
}
if (new File(javaExecutable).exists()) {
logger.debug("Determined java executable: {}", javaExecutable);
} else {
logger.debug("Java executable not found. Trying just java and hope it's in path");
javaExecutable = "java";
}
return javaExecutable;
}
@Data
@ReflectionMarker
@AllArgsConstructor
@EqualsAndHashCode
private static class ExecutedScript {

33
core/src/main/java/org/nzbhydra/database/DatabaseRecreationBean.java Normal file
View File

@ -0,0 +1,33 @@
/*
* (C) Copyright 2023 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra.database;
import org.springframework.beans.factory.InitializingBean;
import org.springframework.core.Ordered;
public class DatabaseRecreationBean implements InitializingBean, Ordered {
@Override
public void afterPropertiesSet() throws Exception {
DatabaseRecreation.runDatabaseScript();
}
@Override
public int getOrder() {
return -100;
}
}

43
core/src/main/java/org/nzbhydra/database/DatabaseRecreationConfig.java Normal file
View File

@ -0,0 +1,43 @@
/*
* (C) Copyright 2023 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra.database;
import com.google.common.collect.Sets;
import org.springframework.boot.sql.init.dependency.AbstractBeansOfTypeDatabaseInitializerDetector;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.util.Set;
@Configuration
public class DatabaseRecreationConfig extends AbstractBeansOfTypeDatabaseInitializerDetector {
@Bean
public DatabaseRecreationBean getDatabaseRecreationBean() {
return new DatabaseRecreationBean();
}
@Override
protected Set<Class<?>> getDatabaseInitializerBeanTypes() {
return Sets.newHashSet(DatabaseRecreationBean.class);
}
@Override
public int getOrder() {
return -1000;
}
}

79
core/src/main/java/org/nzbhydra/database/FlywayMigration.java
View File

@ -1,79 +0,0 @@
/*
* (C) Copyright 2017 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra.database;
import org.flywaydb.core.Flyway;
import org.flywaydb.core.api.FlywayException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.autoconfigure.flyway.FlywayMigrationStrategy;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.sql.SQLException;
@Configuration
public class FlywayMigration {
private static final Logger logger = LoggerFactory.getLogger(FlywayMigration.class);
@Bean
public FlywayMigrationStrategy flywayMigrationStrategy() {
return new FlywayMigrationStrategy() {
@Override
public void migrate(Flyway flyway) {
try {
flyway.migrate();
} catch (FlywayException e) {
if (e.getMessage().contains("1.15")) {
logger.info("Found failed database migration. Attempting repair");
flyway.repair();
try {
flyway.getConfiguration().getDataSource().getConnection().createStatement().executeUpdate("delete from PUBLIC.\"schema_version\" where \"version\" = '1.15' or \"version\" = '1.16'");
} catch (SQLException e1) {
logger.error("Error while deleting old migration steps", e);
}
flyway.migrate();
} else if (e.getMessage().contains("1.21")) {
logger.info("Found failed database migration. Attempting repair");
flyway.repair();
try {
flyway.getConfiguration().getDataSource().getConnection().createStatement().execute("delete from PUBLIC.\"schema_version\" where \"version\" = '1.15' or \"version\" = '1.16'");
flyway.getConfiguration().getDataSource().getConnection().createStatement().executeUpdate("delete from PUBLIC.\"schema_version\" where \"version\" = '1.15' or \"version\" = '1.16'");
} catch (SQLException e1) {
logger.error("Error while deleting old migration steps", e);
}
flyway.migrate();
} else if (e.getMessage().contains("Applied to database : 182559665")) {
//Script had to be changed because with update to SB 2.2 and Flyway 6.0 a comment using // was not properly read in the file V1.0__INITIAL.sql
logger.debug("Reparing changed initial migration SQL checksum");
// flyway.repair();
try {
flyway.getConfiguration().getDataSource().getConnection().createStatement().execute("update \"schema_version\" x set x.\"checksum\" = 1776042577 where x.\"script\" = 'V1.0__INITIAL.sql'");
} catch (SQLException e1) {
logger.error("Error while changing initial migration SQL checksum", e);
}
flyway.migrate();
} else {
throw new RuntimeException("Error while migrating database", e);
}
}
}
};
}
}

69
core/src/main/java/org/nzbhydra/database/H2DialectExtended.java Normal file
View File

@ -0,0 +1,69 @@
/*
* (C) Copyright 2022 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra.database;
import org.h2.engine.Constants;
import org.hibernate.MappingException;
import org.hibernate.dialect.DatabaseVersion;
import org.hibernate.dialect.H2Dialect;
import org.hibernate.dialect.SimpleDatabaseVersion;
import org.hibernate.dialect.sequence.ANSISequenceSupport;
import org.hibernate.dialect.sequence.SequenceSupport;
import org.hibernate.engine.jdbc.dialect.spi.DialectResolutionInfo;
import org.hibernate.tool.schema.extract.internal.SequenceInformationExtractorLegacyImpl;
import org.hibernate.tool.schema.extract.spi.SequenceInformationExtractor;
public class H2DialectExtended extends H2Dialect {
public H2DialectExtended(DialectResolutionInfo info) {
this();
}
public H2DialectExtended() {
//Instance is created without DatabaseVersion info (or ZERO)
super(new SimpleDatabaseVersion(Constants.VERSION_MAJOR, Constants.VERSION_MINOR, Constants.BUILD_ID));
}
public H2DialectExtended(DatabaseVersion version) {
this();
}
@Override
public String toBooleanValueString(boolean bool) {
return bool ? "TRUE" : "FALSE";
}
@Override
public SequenceInformationExtractor getSequenceInformationExtractor() {
return SequenceInformationExtractorLegacyImpl.INSTANCE;
}
@Override
public String getQuerySequencesString() {
return "select * from INFORMATION_SCHEMA.SEQUENCES";
}
@Override
public SequenceSupport getSequenceSupport() {
return new ANSISequenceSupport() {
@Override
public String getSequenceNextValString(String sequenceName) throws MappingException {
return "values next value for " + sequenceName;
}
};
}
}

50
core/src/main/java/org/nzbhydra/database/migration/V2__MOVE_GENERIC_STORAGE.java
View File

@ -1,50 +0,0 @@
/*
* (C) Copyright 2017 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra.database.migration;
import org.flywaydb.core.api.migration.BaseJavaMigration;
import org.flywaydb.core.api.migration.Context;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.config.ConfigReaderWriter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.sql.ResultSet;
import java.sql.Statement;
public class V2__MOVE_GENERIC_STORAGE extends BaseJavaMigration {
private static final Logger logger = LoggerFactory.getLogger(V2__MOVE_GENERIC_STORAGE.class);
@Override
public void migrate(Context context) throws Exception {
ConfigReaderWriter configReaderWriter = new ConfigReaderWriter();
BaseConfig baseConfig = configReaderWriter.loadSavedConfig();
try (Statement statement = context.getConnection().createStatement()) {
try (ResultSet resultSet = statement.executeQuery("select * from GENERIC_STORAGE_DATA")) {
while (resultSet.next()) {
String data = resultSet.getString(2);
String key = resultSet.getString(3);
baseConfig.getGenericStorage().put(key, data);
logger.debug("Migrating GenericStorageData with key {}", key);
}
}
}
configReaderWriter.save(baseConfig);
logger.info("Migrated {} GenericStorageData entries", baseConfig.getGenericStorage().size());
}
}

50
core/src/main/java/org/nzbhydra/database/migration/V3__MOVE_GENERIC_STORAGE.java
View File

@ -1,50 +0,0 @@
/*
* (C) Copyright 2017 TheOtherP (theotherp@posteo.net)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.nzbhydra.database.migration;
import org.flywaydb.core.api.migration.BaseJavaMigration;
import org.flywaydb.core.api.migration.Context;
import org.nzbhydra.config.BaseConfig;
import org.nzbhydra.config.ConfigReaderWriter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.sql.ResultSet;
import java.sql.Statement;
public class V3__MOVE_GENERIC_STORAGE extends BaseJavaMigration {
private static final Logger logger = LoggerFactory.getLogger(V3__MOVE_GENERIC_STORAGE.class);
@Override
public void migrate(Context context) throws Exception {
ConfigReaderWriter configReaderWriter = new ConfigReaderWriter();
BaseConfig baseConfig = configReaderWriter.loadSavedConfig();
try (Statement statement = context.getConnection().createStatement()) {
try (ResultSet resultSet = statement.executeQuery("select * from GENERIC_STORAGE_DATA")) {
while (resultSet.next()) {
String data = resultSet.getString(2);
String key = resultSet.getString(3);
baseConfig.getGenericStorage().put(key, data);
logger.debug("Migrating GenericStorageData with key {}", key);
}
}
}
configReaderWriter.save(baseConfig);
logger.info("Migrated {} GenericStorageData entries", baseConfig.getGenericStorage().size());
}
}

42
core/src/main/java/org/nzbhydra/debuginfos/DebugInfosProvider.java
View File

@ -1,9 +1,14 @@
package org.nzbhydra.debuginfos;
import com.fasterxml.jackson.core.JsonProcessingException;
import jakarta.annotation.PostConstruct;
import jakarta.persistence.EntityManager;
import jakarta.persistence.PersistenceContext;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.EqualsAndHashCode;
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.tuple.Pair;
import org.javers.core.JaversBuilder;
import org.javers.core.diff.Diff;
import org.nzbhydra.Jackson;
@ -17,6 +22,7 @@ import org.nzbhydra.logging.LogAnonymizer;
import org.nzbhydra.logging.LogContentProvider;
import org.nzbhydra.logging.LoggingMarkers;
import org.nzbhydra.problemdetection.OutdatedWrapperDetector;
import org.nzbhydra.springnative.ReflectionMarker;
import org.nzbhydra.update.UpdateManager;
import org.nzbhydra.webaccess.HydraOkHttp3ClientHttpRequestFactory;
import org.nzbhydra.webaccess.Ssl;
@ -26,12 +32,10 @@ import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.actuate.management.ThreadDumpEndpoint;
import org.springframework.boot.actuate.metrics.MetricsEndpoint;
import org.springframework.core.env.ConfigurableEnvironment;
import org.springframework.stereotype.Component;
import org.springframework.transaction.annotation.Transactional;
import javax.annotation.PostConstruct;
import javax.persistence.EntityManager;
import javax.persistence.PersistenceContext;
import java.io.File;
import java.io.FileOutputStream;
import java.io.FileReader;
@ -47,6 +51,7 @@ import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.Set;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
@ -82,6 +87,9 @@ public class DebugInfosProvider {
private EntityManager entityManager;
@Autowired
private OutdatedWrapperDetector wrapperDetector;
@Autowired
private ConfigurableEnvironment environment;
@Autowired
private Ssl ssl;
@ -170,6 +178,20 @@ public class DebugInfosProvider {
return timeAndThreadCpuUsagesList;
}
public static Pair<String, String> getVersionAndBuildTimestamp() {
final Properties properties = new Properties();
try {
properties.load(DebugInfosProvider.class.getResourceAsStream("/config/application.properties"));
} catch (Exception e) {
try {
properties.load(DebugInfosProvider.class.getResourceAsStream("/application.properties"));
} catch (Exception ex) {
throw new RuntimeException("Unable to load application properties", ex);
}
}
return Pair.of(properties.getProperty("build.version"), properties.getProperty("build.timestamp"));
}
private double getUpTimeInMiliseconds() {
return metricsEndpoint.metric("process.uptime", null).getMeasurements().get(0).getValue() * 1000;
}
@ -213,7 +235,7 @@ public class DebugInfosProvider {
logger.info("Metrics:");
final Set<String> metricsNames = metricsEndpoint.listNames().getNames();
for (String metric : metricsNames) {
final MetricsEndpoint.MetricResponse response = metricsEndpoint.metric(metric, null);
final MetricsEndpoint.MetricDescriptor response = metricsEndpoint.metric(metric, null);
logger.info(metric + ": " + response.getMeasurements().stream()
.map(x -> x.getStatistic().name() + ": " + formatSample(metric, x.getValue()))
.collect(Collectors.joining(", ")));
@ -265,7 +287,12 @@ public class DebugInfosProvider {
}
public void logThreadDump() {
try {
//Fails on native image
logger.debug(threadDumpEndpoint.textThreadDump());
} catch (Exception e) {
logger.error("Unable to create thread dump : {}", e.getMessage());
}
}
private String formatSample(String name, Double value) {
@ -362,6 +389,7 @@ public class DebugInfosProvider {
}
@Data
@ReflectionMarker
public static class TimeAndThreadCpuUsages {
private final Instant time;
private final List<ThreadCpuUsage> threadCpuUsages = new ArrayList<>();
@ -372,16 +400,22 @@ public class DebugInfosProvider {
}
@Data
@ReflectionMarker
@AllArgsConstructor
public static class ThreadCpuUsage {
private final String threadName;
private final long cpuUsage;
}
@EqualsAndHashCode(callSuper = true)
@Data
@ReflectionMarker
public static class DiffableCategoriesConfig extends CategoriesConfig {
private Map<String, Category> categoriesMap = new HashMap<>();
public DiffableCategoriesConfig() {
}
public DiffableCategoriesConfig(CategoriesConfig categoriesConfig) {
categoriesConfig.getCategories().forEach(x -> {
categoriesMap.put(x.getName(), x);

12
core/src/main/java/org/nzbhydra/debuginfos/DebugInfosWeb.java
View File

@ -7,9 +7,11 @@ import lombok.AllArgsConstructor;
import lombok.Data;
import org.nzbhydra.GenericResponse;
import org.nzbhydra.NzbHydra;
import org.nzbhydra.config.BaseConfigHandler;
import org.nzbhydra.config.ConfigProvider;
import org.nzbhydra.logging.LogContentProvider;
import org.nzbhydra.logging.LogContentProvider.JsonLogResponse;
import org.nzbhydra.springnative.ReflectionMarker;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
@ -52,6 +54,8 @@ public class DebugInfosWeb {
private MappingsEndpoint mappingsEndpoint;
@Autowired
private ConfigProvider configProvider;
@Autowired
private BaseConfigHandler baseConfigHandler;
private static final Logger logger = LoggerFactory.getLogger(DebugInfosWeb.class);
@ -179,7 +183,7 @@ public class DebugInfosWeb {
final List<RequestMappingConditionsDescription> conditionsDescriptions = ((Map<String, List<DispatcherServletMappingDescription>>) mappingsEndpoint.mappings().getContexts().get("NZBHydra2").getMappings().get("dispatcherServlets")).get("dispatcherServlet")
.stream().filter(x1 -> x1.getHandler().contains("nzbhydra"))
.map(x -> x.getDetails().getRequestMappingConditions()).collect(Collectors.toList());
.map(x -> x.getDetails().getRequestMappingConditions()).toList();
final List<PrefixAndEndpoint> prefixAndEndpoints = new ArrayList<>();
final Multimap<String, Endpoint> endpoints = HashMultimap.create();
@ -212,12 +216,13 @@ public class DebugInfosWeb {
final String msg = "Set log file level to debug and enabled the following logging markers: " + markersToEnable;
logger.info(msg);
configProvider.getBaseConfig().save(true);
baseConfigHandler.save(true);
return ResponseEntity.ok(msg);
}
@Data
@ReflectionMarker
public static class ThreadCpuUsageChartData {
private final String key;
private final List<TimeAndValue> values;
@ -234,6 +239,7 @@ public class DebugInfosWeb {
}
@Data
@ReflectionMarker
@AllArgsConstructor
public static class TimeAndValue {
private final Instant time;
@ -241,6 +247,7 @@ public class DebugInfosWeb {
}
@Data
@ReflectionMarker
@AllArgsConstructor
public static class PrefixAndEndpoint {
private final String prefix;
@ -249,6 +256,7 @@ public class DebugInfosWeb {
}
@Data
@ReflectionMarker
@AllArgsConstructor
public static class Endpoint {
private final String endpoint;

6
core/src/main/java/org/nzbhydra/downloading/DownloadResult.java
View File

@ -17,7 +17,8 @@
package org.nzbhydra.downloading;
import lombok.Data;
import org.nzbhydra.searching.dtoseventsenums.SearchResultItem;
import org.nzbhydra.config.downloading.DownloadType;
import org.nzbhydra.springnative.ReflectionMarker;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.HttpHeaders;
@ -30,6 +31,7 @@ import java.util.regex.Matcher;
import java.util.regex.Pattern;
@Data
@ReflectionMarker
public class DownloadResult {
private static final Logger logger = LoggerFactory.getLogger(DownloadResult.class);
@ -65,7 +67,7 @@ public class DownloadResult {
protected String getFileName() {
String filename = title;
if (downloadEntity.getSearchResult().getDownloadType() == SearchResultItem.DownloadType.NZB) {
if (downloadEntity.getSearchResult().getDownloadType() == DownloadType.NZB) {
filename += ".nzb";
} else {
filename += ".torrent";

24
core/src/main/java/org/nzbhydra/downloading/FileDownloadEntity.java
View File

@ -1,27 +1,43 @@
package org.nzbhydra.downloading;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import jakarta.persistence.Column;
import jakarta.persistence.Convert;
import jakarta.persistence.Entity;
import jakarta.persistence.EnumType;
import jakarta.persistence.Enumerated;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.Index;
import jakarta.persistence.ManyToOne;
import jakarta.persistence.SequenceGenerator;
import jakarta.persistence.Table;
import lombok.Data;
import org.hibernate.annotations.OnDelete;
import org.hibernate.annotations.OnDeleteAction;
import org.nzbhydra.config.SearchSource;
import org.nzbhydra.config.downloading.FileDownloadAccessType;
import org.nzbhydra.searching.db.SearchResultEntity;
import org.nzbhydra.searching.searchrequests.SearchRequest.SearchSource;
import org.nzbhydra.springnative.ReflectionMarker;
import org.nzbhydra.web.SessionStorage;
import javax.persistence.*;
import java.time.Duration;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
@Data
@ReflectionMarker
@Entity
@Table(name = "indexernzbdownload", indexes = {@Index(name = "NZB_DOWNLOAD_EXT_ID", columnList = "EXTERNAL_ID")})
public class FileDownloadEntity {
public final class FileDownloadEntity {
@Id
@GeneratedValue(strategy = GenerationType.AUTO)
protected int id;
@SequenceGenerator(allocationSize = 1, name = "INDEXERNZBDOWNLOAD_SEQ")
private int id;
@ManyToOne
@JsonIgnoreProperties(value = {"handler", "hibernateLazyInitializer"})
@OnDelete(action = OnDeleteAction.CASCADE)
private SearchResultEntity searchResult;
@Enumerated(EnumType.STRING)

Some files were not shown because too many files have changed in this diff Show More